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PREFACE 



This report is intended for practitioners, researchers, and poticymakers concerned 
with current performance-assessment efforts and their effects on instructional quality. It 
describes the results of an evaluation of Vermont's statewide assessment initiative, a 
program that has garnered widespread attention nationwide because of its reliance on 
portfolios of student work This report provides information about the implementation 
of the program, the effects of the reform on educational practice, the analytic challenges 
presented by the portfolio scoring process, the reliability and validity of portfolio scores, 
and the tensions between assessment and instructioruil reform. The Vermont 
experience has important implications for reforms that are underway or under 
consideration in other jurisdictions. 

This project was conducted by RAND under the auspices of the Center for 
Research on Evaluation Standards and Student Testing, The report presents the results 
of RAND's evaluation of the first statewide implementation of the Vermont portfolio 
assessment program, which was tmdertaken in the 1991-92 school year. This report 
integrates and adds to material presented in earlier reports and conference papers issued 
by the RAND study team during 1991-91 

Tlie first four authors contributed to the preparation of all sections of this report. 
However, Brian Stecher was the principal author of Chapter 2. and Stephen Klein took 
primary responsibility for preparing Chapters 3 and 4, Dan Koretz was the principal 
author of the remaining chapters, Dan McCaffrey performed statistical arwlyses that are 
reflected throughout the report. Ed Deibert was rebpomible for many of the operational 
aspects of the study and also contributed material reflected in Chapter 2. 
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SUMMARY 



Since 1988, Vermont has been developing an innovative assessment program in 
which student portfolios play a central role. Mathematics and writing portfolios were 
piloted in selected and volunteer schools in 1990-91, and the program was first 
implemented statewide in the 1991-1992 school year. RAND has consulted wilh 
Vermont about the development of the program since 198r and has been evaluating the 
program since 1990 as part of the Center for Research on Evaluation, Standards, and 
Student Testing. RAND's evaluation has encompassed the implementation of the 
program, its effects on educational practice, and the quality of the achievement data 
produced by the program. 

This report describes the result of " \NUs evaluation of the first statewide 
implementation of the program in 1991-92. This report integrates and adds to material 
presented in earlier interim reports (Koretz, Stecher, and Deibert, 1992; Koretz, 
McCaffrey, Klein, Bell, and Stecher, 1992) and in conference papers. 

BACKGROUND ON THE VERMONT ASSESSMENT PROGRAM 

Until recently, Vermont had no statewide assessment program. By the late 1980s, 
however, pressure began building to create one. Richard Mills, the Commissioner ot 
Education, and W. Ross Brewer, thm Director of Policy and Planning for the State 
Education Department, irxitiated development of the current system in 1988. The 
program, like many other performance assessment initiatives nationwide, was intended 
to serve two purposes: (1) to provide useful information about student performance, and 
(2) to encourage improvement in teaching. Mills' and Brewer's goal was to build a 
system that would be consistent with Vermont's tradition of highly decentralized 
educational decision making while still encouraging "a high contmon standard of 
achievement for all students'* (Mills and Brewer, 1988). They explicitly acknowledged 
that the system would necessarily go beyond current technologies and would require a 
long period of development. 

The system that emerged has as its centerpiece portfolios of student work that are 
collected by classroom teachers but are scored using criteria that are consistent across 
the state. The guidelines for the operation of the portfolio program and the criteria for 
scoring student work have been developed by committees consisting largely of 
volunteer teachers. Teachers are given only limited gviidebnes for the compilation of 
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portfolios-for example, the riimber of pieces that should be included and the broad 
categories of work from which the pieces should be drawa The portfolios are 
complemented by a system of "uniform tests ' that are standardized in terms of content 
and administrative conditions but are not restricted to the multiple-choice format. 
Unlike the portfolios, the uniform tests have been developed largely by external 
contractors. Although the assessment system is intended to cover many subjects, at 
present it is limited to writing and mathematics in grades 4 and 8. 

In mathematics, the system (as implemented in 1991-92) required students and 
teachers to construct a portfolio for each student comprising five to seven "best pieces. " 
Tl\ese pieces were to be of three types: puzzles, applications, and investigations. A 
sample of portfolios from each school was submitted to one of seven regional meetings 
for scoring by other teachers. Each piece was scored on seven dimensions, four 
pertaining to problem-solving and three pertaining to communication. All dimensions 
were scored on four-point analytic scales. Composite scores (across the best pieces) 
were calculated for each dimension, but the state did not report a total score (combined 
across dimensions). A subsample of portfolios was scored twice for analysis of 
reliability. 

The writing portfolio system operated differently. Students were asked to include 
in their portfolios six to eight pieces (depending on their grade level) from several 
different categories of work- (These categories were broader than traditional genres; 
examples include "a poem, short story, or personal narration" and "a personal response 
to a book, event, current issue, mathematical problem, or scientific phenomenon.") The 
student had to designate one piece from any category as the "best piece." The best piece 
was scored on five dimensions; again, all five were four-point analytic scales. Raters 
were asked to score the remaining pieces in each portfolio as a set on each dimension 
and were not required to score each piece separately in arriving at scores for the set. The 
initial rater was the student's own classroom teacher, and a sample of portfolios was 
submitted for rescoringby a second rater. 

IMPLEMENTATION AN*? OVIPACT 

In 1991-92, information about implementation and impact was gathered from 
teachers and principals. Teachers responded to a questionnaire that focused on the 
mathematics portfolios. Their responses indicated that participation in the mathematics 
portfolio assessment was extensive and that the program ap(>ears to have had a positive 
impact on instruction. 



• Virtually all fourth and eighth grade math teachers received state-sponsored 
training in the use of portfolios. They generally rated this training as effective. 

• Nearly all fourth and eighth grade students compiled portfolios of their 
mathematics work. 

• Teachers reported devoting substantially more attention to problem solving and 
commtmication in teaching mathematics as a result of the program. 

• Teachers reported some changes in mathematics instructional practices; for 
example, students spent more time working in small groups and in pairs. 

The mathematics portfolio assessment program produced other benefits as well. 
Portfolios were an impetus for jjositive changes in teacher and student attitudes 
regarding mathematics and learning. Many teachers reported that both they and 
students were generally more enthusiastic about mathematics as a result of the 
portfolios. The portfolios also provided teachers with new perspectives on students' 
abilities. 

However, there were also shortcomings in implementation, as might be expected 
with a reform of this scale and novelty. The most serious problem was continuing 
confusion on the part of many teachers about the purposes of the mathematics portfolios 
and the proper practices to use to implement the assessment system. Most teachers felt 
they were unprepared to use the portfolios on at least some occasions. Teachers also 
raised concerns about the lack of information from the state and the rapid speed of the 
reform. 

In addition, the decentralized nature of the reform encouraged teachers to adopt 
idiosyncratic practices regarding mathematics portfolios. Teachers assigned tasks of 
varying rKJvelty and complexity. They adopted different rules regarding revisior>s and 
the amount of assistance pupils could receive when preparing portfolio pieces. These 
contextual differences probably affected student performance, and, as a result, the 
validity of comparisor>s based on portfolio scores. 

Principals echoed these concerns and raised other issues relevant to both the 
writing arKl mathematics portfolios. They noted that the positive changes came at a 
price. The writing and mathematics portfolios required a sizable commitment of 
preparation and classroom time from teachers. The program also posed sizable 
demands on school resources for release time and substitute teachers. Furthermore, the 
portfolio assessment generated some negative attitudes on the part of teachers and 
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principals. Both groups perceived the time and resources demands to be burdensome. 
Teachers also were anxious about scoring and the potential uses of scores, particularly 
for comparative purposes. 

Although Vermont educators varied markedly in their opinions of the assessment 
program, a widespread view was that it was a worthwhile burden That is, the demands 
of the program were seen as large and burdensome, but support for the program as a 
tool of instructional reform was high nonetheless. Perhaps the most telling sign of 
support was the fact that in roughiy half of the schools in which we conducted 
interviews, the use of portfolios had already been expanded in some form beyorui the 
grades or subjects included in the state program. 

Overall, Vermont appears to be building momentum toward greater acceptance 
and more efficient use of the portfolios by responding to needs with targeted training. 
However, there are still important gaps between the ideal and the reality. The most 
sigTuficant gaps relate to standardization of basic practices and the establishment of a 
common understanding of fundamental constructs, such as problem solvir\g and 
mathematical commurucation. 

RELIABILITY OF PORTFOLIO SCORES FOR INDIVIDUALS 

Although the Vermont program was not designed to produce student- level scores 
for external use (that is, for use outside of students' own schools), the scores assigned to 
students are the basic building block of the system. The quality of student-level scores 
influences the adequacy of aggregate scores and limits the uses to which the assessment 
results can be put. 

An essential component of reliability is "rater reliability ' or "rater agreement' -that 
is, the extent to which different raters agree about the score that should be assigned to a 
given piece of work. The rater reliability of portfolio scores in both mathematics and 
writing was very low. The Vermont system was designed to provide separate scores on 
each scorir^ dimensioa For individual pieces on a single dimension, the correlations of 
scores between raters ranged from .28 to .57; the average correlatior\s (across all 
dimensions) ranged from .34 to 43. (A correlation of 1. 00 indicates perfect agreement, 
while 0.00 indicates a total lack of a systematic relationship between raters.) The 
percentage of cases in which raters agreed on a score was generally not much higher 
than expected by chance. Summing across all pieces within a portfolio on a single 
scoring dimension raised the average reliability coefficients only slightly, to a range of 
.38 to .49. Vermont did not combine scores across dimensions because to do so would 
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have been inconsistent with the program's goal of providing dimension-level scores as 
instructional feedback. However, even summing across dimensions as well as pieces to 
get a single total score would have raised the average reliability coefficients only to a 
range of .49 to .60. 

QUALITY OF AGGREGATE SCORES 

The Vermont system is intended to provide various types of aggregate scores, 
including the proportion of students statewide reaching each score point on each scoring 
dimension and comparative data about districts or schools. 

In 1991-92, nuny of the aggregate statistics the state wished to report were not of 
high enough quality to use, aiKi the State Department of Education accordingly declined 
to release them. The low qiaality of aggregate scores stemmed in part fro n\ the low rater 
reliability of the score« assigned to iiKlividual students. 

Because of the relatively large number of students involved, statewide average 
scores on each dimension were quite reliable despite the low reliability of scoring. For 
example, in the fourth grade, the average score (on a scale from 1 to 4^^ cn the dimension 
"math representations" was 2.3; the confidence band for this average was approximately 
2.25 to 2.35. However, this confidence band takes into account only sampling error and 
the reliability of ratings. It does T)ot consider other factors that might threaten the 
reliability of scores, such as limited genera lizability of performance across the small 
sample of tasks scored for each student. 

On the other hand, school-level average scores were too unreliable to use. The 
unreliability of scoring contributed to this problem but was not its main cause. Rather, 
the unreliability of school averages stennmed mainly from the small size of n\any 
Vermont schools, particxilarly in the fourth grade. 

Furthermore, the proportion of students reaching each level could not be reported 
even for the state as a whole. One effect of unreliable scoring is to spread out the 
distribution of scores, so that too many students receive high and low scores arid too few 
receive scores near the middle of the distribution The resulting bias in estimates of the 
proportion of students receiving each score was large but could not be estimated well 
enough to permit statistical correctioa 

VALIDITY 

Rater reliability is an essential but iasufficient criterion for evaluatir^ assessment 
results. The ultimate question is the validity of the results-that is, the extent to which 
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they support the conclusions people base on them. Unreliable scoring undermines 
validity, but reliable scoring does not guarantee it. 

The issue of validity was largely moot in 1991-92 because of the low rater 
reliability of scores, but we explored what conclusions about validity might have been 
warranted if reliability of scoring had been better. This was done by "disattenuating ' 
relationships m the data to estimate what would be found if scorit^ were perfectly 
reliable-a conventional approach, but risky in the case of Vermont's data because the 
nujasurement error was very large. Hernre oiu* findings can be considered only 
expbratory. 

In general, our analyses of validity evidernre were unpersuasive. A few of the 
patterns one would expect did appear in the data; for example, disattenuated 
correlations between writing portfolio scores and scores on the uniform test of writing 
were consistent with other assessments. However, many of the expected relationships 
failed to appear. For example, mathematics portfolio scores would be expected to 
correlate more highly with scores on the mathematics uniform test than with scores on 
the writing uniform test. In the maia this was mi the case. Perhaps more important, 
evaluation of validity was hampered by the lack of a sufficiently clear definition of the 
attributes the portfolios are interxied to measure. Without such a defir\ition^ one cannot 
adequately use other information about performarKe to gauge the validity of scores. 

Oxu'data about program implementation also raises concerns about validit)*. 
Teachers reported large variatior\s in key aspects of implennentation, such as policies 
pertaining to revision of students' products. These var ations in implementation might 
mask differences in student's capabilities or generate spurious differences where rv^ne 
are warranted, 

IMPLICATIONS 

The Vermont assessmer\t program has attracted nationwkie attention A critical 
question for observers from outside Vermont is the extent to which the findings 
described here have implications for their current or planned perfonnanoe-assessment 
programs. 

The Vermont program is in some respects unusual. Vermont's program 
emphasizes portfolios, which are unstandardized performances, while many other 
cxu'rent programs rely on standardized performances, such as uiuform writing prompts 
or standardized open-ended mathematics questions. Moreover, the Vermont program, 
like any other, has idiosytKralic elements (such as the particular rubrics used in scoring). 
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Nonetheless, the Vermont program has much in common with many other current 
performance-assessment programs, and the Vermont experience has important 
implications for them. It shares with many other programs the dual fundamental goals 
of measuring student performance and sparking improvements in instruction/ and the 
Vermont program's specific instructional objectives (such as an increased emphasis on 
problem solving in mathematics) are shared by programs nationwide. Moreover, 
leading proposals would create new programs that resemble the Vermont program. For 
example, the New Standards Project has long called for extensive reliance on 
unstandardized performances. The Vermont experience can provide guidance for the 
design of these programs and can help set expectations for their performance. 

Expectations for quality of meararement In 1991-92, the quality of the data 
about student performance yielded by the portfolio program was so low that it severely 
restricted the appropriate uses of scores. Although unique aspects of the Vermont 
program undoubtedly contributed to this problem, some of the likely causes will arise in 
other programs as well For example, there are at least three factors that arejikrty^ 
contributors to the unreliability of scoring: inadequate scoring rubrics, insufficient 
training, and the lack of starKlardization of tasks, Unstatvdardized tasks are likely to . 
complicate scoring substantially in most programs that rely on them, particularly in ^. 
subjects such as mathematics arvd scierxre in which tasks are likely to vary greatly. 
Training of teachers and raters is likely to be an arduous task in many large«scale 
systems, particularly those that aim to change instruction markedly. Similarly, the 
limited generaiizability of performance across tasks that we found in the mathematics 
portfolios is consistent v^th findings from diverse performance assessments, both 
standardized and not, and threaterw the validity of IixfererKes based on small numbers 
of tasks. The wide variations in program implementation we found are likely to occxar in 
other programs that attempt to integrate assessment and instruction and will undermine 
the validity of many irUeruied uses of scores. 

Expectations for impact on educational practice. The Vermont experience 
provides grounds for optimism that performance assessment programs may be able to 
ir\fluence instruction substantially. At the same time, if offers reasons to temper the 
optimism that is currently so widespread. Even after years of development, a year of 
piloting, and a full year of statewide implementation, the program had made only 
partial strides toward its instructional goals. For example, it had not yet provided 
mathematics teachers with a sufficiently consistent interpretation of performance goals. 
Moreover, the partial success attained to date has come at a high price in time, stress. 
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and money. It seems clear that the performance assessment itself is only part of what is 
needed; it must be coupled, for example, with exterxsive and continuing professional 
development. 

Tensions between the goals of assessment programs* Many performance- 
assessment programs and proposals share with the Vermont program the dual goals of 
improving instruction aiKl measuring student perfonnatvce. To some degree, those two 
goals conflict. For example, starvdarJization of both tasks and admirustrative conditions 
will generally improve the quality of student performance data, but standardization 
may hinder the integration of assessment with instruction and lessen teachers* feelings 
of ownership and commitment. The improvement of these programs will often require 
not only that steps be taken to lessen the tensions between these goab, but also that 
educators make difficult compromises between them. That is, they may need to choose 
how much of a decrement in measurement quality they are veiling to accept to facilitate 
instructional improvement, and vice versa. 

This fundamental tension manifested itself concretely in the Vermont program. 
We noted above that the wide variations in the implementation of the program, such as 
variatior\s in policies toward the revision of students' work, threaten the validity of 
comparisons based on portfolio scores. To some degree, however, these variatiorxs in 
practice may be instructionally beneficial. For example, many teachers would argue that 
as students become more able, they should become m< autonomous and should 
receive less assistance and structure to guide their revisions. To eliminate such 
variatiorxs may be instructionally urKiesirable, but to leave them in place undermines the 
validity of comparisons based on portfolio scores. The ter\sion between the program's 
two fundamental goals also marufested itself in designing the scoring system. For the 
sake of professional development, Vermont opted to train all teachers to score and to use 
all teachers who volunteered as raters. To increase reliability, however, one would want 
to limit scoring to the number of raters that can be trained to a high level of accuracy. 

Requirements for evaluation. Programs that hold people accountable for scores 
on assessments are not self-evaluating. That is, increases in scores on the assessments 
for which people are held accountable are not sufficient to indicate that the programs are 
meeting their goals. This fact is now widely recognized in the case of programs using 
traditional, multiple-choice tests, but it is no less true of performance-assessment 
programs. These new programs are intended to have pervasive effects throughout the 
educational system, and evaluating these varied effects requires collecting information 
about a wide range of questiorxs. 
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Explicit investigation of program implementation-and variations in 
implementation-is important to provide corrective feedback and to explore matters of 
equity. Changes in instruction must be investigated directly and cannot be inferred 
adequately from performance on the assessments. Examination of reliability must go 
well beyond simple rates of agreement among raters. It will be important to explore 
both the causes and the effects of imreliable scoring, when it is found. If and when 
reliable scoring is attained, it will be essential to research score reliability-that is, the 
generalizability of performance across tasks (and other facets) as well as raters-and 
other evidence of validity. Finally, to validate the new assessments and to obtain solid 
estimates of changes in student learning, evaluators will need to administer additional 
measures of the domains that the new assessments purport to measure, 

CONCLUSIONS 

The experience of the Vermont portfolio program to date suggests the need for 
patience, moderate expectations, and ongoing, formative evaluation. Neither of the 
basic goals of the program-using complex performances to measure student 
performance and utilizing a performance-assessment program to spur instructional 
improvement- can be met easily or quickly. Moreover, the Vermont experience 
illustrates the tensions between these goals and the need to make difficult trade-offs in 
compromising between them. 
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1. INTRODUCTION 



Since 1988, Vermont has been developing an assessment program that is at the 
cutting edge of innovation in large-scale assessments. Although a rapidly growing 
number of statewide assessment programs iiKorporate some form of performance 
assessment, the Vermont program is unusual among them in that its centerpiece is 
student portfolios and "best pieces" drawn from them. Portfolio-based assessment is not 
new, but Vermont was first state to make portfolios the backbone of a statewide 
assessment program. The Vermont program is also unusual in the degree to which it is 
"bottom up:" many aspects of the assessments in each subject are worked out iteratively 
by committees of teachers, and classroom teachers retain wide latitude in implementing 
the program. 

The Vermont Department of Education selected 48 schools to pilot fourth- and 
eighth-grade assessments in writing and mathematics during the 1990-91 school year, 
and 90 other schools asked to participate to varying degrees in the pilot efforts that year. 
The first state-wide implementation of the assessment in those two grades and subjects 
was conducted in the 1991-92 school year. 

RAND has corxsulted with Vermont about the development and eventual 
evaluation of the assessment program since August, 1988. Since 1990, RAND, as part of 
the Center for Research on Evaluation, Standards, and Student Testing (CRESST), has 
been carrying out a multi-faceted evaluation of the assessment program and its effects. 
This monograph reports the findings of the RAND/CRESST study through the 1991-92 
school year. 

BACKGROUND ON THE VERMONT ASSESSMENT PROGRAM 

Until recently, Vermont had no regular statewide assessment program. By the late 
1980s, however, pressure was building to provide regular information on student 
performance, and by 1988, the state Department of Education began movement toward 
establishment of a statewide assessment system. 

The deliberations that led to the decision to build the present, portfolio-based 
system are difficult to summarize succinctly because they were lengthy and involved 
many diverse people, including the Commissioner of Education {Richard Mills), the 
Department's then-Director of Policy and Plannit^ (Ross Brewer), the governor, 
members of the state board, local board members, teachers, and others. Several 
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persistent themes, however, were stressed by Mills, Brewer, and others working to build 
the system.^ Ideally, the new system would: 

• Avoid the distortions of educational practice that conventional test-based 
accountability appeared to have created in some other states; 

• Encourage good practice and be integrally related to the professional 
development of educators; 

• Reflect the Vermont tradition of local autorujmy, "encourage local 
inventiveness, [and] preserve local variations in curriculum and approach to 
teaching" (Mills and Brewer, 1988, pp. 3, 5); 

• Provide '*a high common standard of achievement for all students,*^' and 

• Encourage greater equity in educational opporttinity (Milk and Brewer, 1988, 
p. 3). 

Those responsible for the nascent program were aware of the difficulties inherent 
in having an assessment program serve many functions at once and had been warned 
that some of their goals for the program pointed to different assessment designs. For 
example, a system designed to provide rich information about students and positive 
incentives for teachers might look very different from a system that was designed 
primarily to provide highly comparable information across schools.^ The system was 
interKled to be a compromise among the many goals for the system; for example, it 
should provide reasonable comparability across schools, but not at the cost of stifling 
good practice and local innovation. 

The basic outline of the assessment program emerged quite quickly. Eventually, 
the assessment would span a broad range of subjects, but the state decided to begin with 
assessments in writing and mathematics in grades 4 and 8. The assessment would have 
three components: year-long student portfolios, "best pieces*' drawn from the portfolios, 
arul state-sponsored "uniform tests." 

^This description is baaed in large part on the first author's participation in meetings and 
discussions with C^partment of Education staff and other3 involved in building the assessment 
program. No single source summarizes the development of the program, but many of the points 
noted here have been described elsewhere. See, for example, Vermont State Department of 
Education, 1990, 1991a, 1991b, and Mills and Brewer, 1988.) 

^Dan Koretz, presentation to the Commissioner Mills, Governor Kunin,and others, 
August, 1988. 
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The details of the program, however, have been worked out only gradually. In 
contrast to the many states that eitlier buy off-the-shelf lests or contract to have new tests 
built on a short schedule, the Vermont program was seen from the outset as a long-term 
and decentralized development effort. For example, in 1988, Mills and Brewer called for 
mixing state-of-the-art assessment techniques with "emerging" techniques and warned 
that the development of the new program would be "a very long effort" (Mills and 
Brewer, 1988). Thus, in both subjects, the so-called "pilot" implementation in 1990-91 
was less a true pilot of a developed program than an integral part of the development 
effort. Indeed, in mathematics, even the first full statewide implementation in the 1991- 
92 school year, the evaluation of which is reported here, would most accurately be 
categorized as a combination of a developmental effort and a pilot test, rather than as an 
initial implementation of a fully planned program. Some of the details of the scoring of 
best pieces in the 1991-92 statewide implementatioa for example, were not resolved 
tmtil spring of 1992, and ratings of entire portfolios have not yet been attempted on a 
large scale. 

Primary responsibility for the development of the portfolio and best-pieces 
components of the program was given to state-sponsored committees of teachers. These 
connmittees worked independently of each other, so the program evolved differently in 
writing and mathematics. 

MATHEMATICS 

As implemented in 1991-92, the mathematics program required that students and 
teachers cull from each students' portfolios a set of five to seven "best pieces." Teachers 
were requested to include in this set of best pieces exemplars of three types of problems: 
puzzles, investigations, and applications. According to the Teacher's Guide (Vermont 
State Department of Education^ 1991c), puzzles are tasks that "require students to identify 
and explore approaches to non-routine problems... [where] most uf the problem deals 
with logic and reasoning." For example, "With a seven minute hourglass and an eleven 
minute hourglass, how could you time the boiling of an egg for 15 minutes?" 
Applications, by comparison, "require students to apply knowledge they already 
possess," For example, "A mature tree can utilize 13 lbs. of carbon dioxide in a year The 
average car spews out 40(X} lbs. of carbon dioxide in a ye ir. How many mature trees 
would you need to utilize this much carbon dioxide?" (The problem continues, 
exploring the relationship of fuel efficiency and carbon dioxide emissions.) Invesiigaiions 
include 'explorations, data collection and analysis or some level of research that leads to 
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conclusions." For example, "Given a piece of graph paper, determine the size of a square 
to be cut out of each comer of the graph paper that will allow you to fold the graph 
paper into an open box with the greatest volume. Try squares from 1 to 9 units. The 
determine the volume of each open box after you have folded the paper. Collect the 
data in a data table of your design. Graph the results. Write up your results." The 
Resource Book provided to each teacher (Vermont State Department of Education, 1991d) 
contained many samples each type of task.^ 

The best-pieces sets of a sample of students from each participating classroom 
were sent to one of seven central sites for scorii\g by groups of volunteer teachers. (The 
samples from each class were selected by the state's assessment contr?ictor, rot the 
classroom teachers.) Training in scoring had been offered state-wide before the scoring 
sessior\s, but the volunteer raters varied substantially in their level of training, and 
supplementary training was provided at the beginning of each three-day scoring 
session. 

AU of the best pieces were graded on four-point scales on seven dimensions; four 
classified as problem-solving and three as communication. The seven dimensions, and 
the scale points on each for individual pieces, were as follows: 

• Understanding the task; 1 . totally misurrferstood; 2. partially understood; 3. 
understood; 4. generalized, applied, extended. 

• How - approaches/procedures: 1. Inappropriate or unworkable; 2. 
appropriate some of the time; 3. workable; 4. effident or sophisticated. 

• Why - decisions along the way: 1. no evidence; 2. possible; 3. inferred with 
certainty; 4. shown/explicated. 

• What - outcomes of activities: 1. no extension; 2. observations; 3. 
connections, applicatior\s; 4. synthesis, generalization, abstraction. 

• Language: 1. No or inappropriate; 2. some of the time; 3. most of the time; 4. 
rich, precise, elegant. 

• Representation: 1. No use; 2. use; 3. accurate and appropriate; 4. perceptive. 

• Presentation: 1. not clear; 2. some clear parts; 3. mostly clear; 4. clear. 



^The Resource Book, which was prepared by a committee of teachere and distributed by the 
state Education Department, contains a collection of activities appropriate for inclusion in 
mathematics portfolios. 
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The ratings on the individual pieces were then aggregated to provide an overall 
rating of the entire set of best pieces on each of ^he seven dimensions. However, the 
nature of the scales for individual pieces led the mathematics committee to reject a 
simple averagir\g across pieces. A set of three 4s ("perceptive") and three 2s ("use") on 
"Representation" do not average to a 3 ("accurate and appropriate"). Accordingly, the 
mathematics committee developed a "holistic" rating system in which the 4 scale points 
for the aggregate rating were defined differently than those for the individual ratings, 
and a fixed algorithm (based on the number of instances of different scores on the 
individual pieces) was used to create the aggregate scores. 

This system for aggregating scores was too detailed to warrant a full description 
here, but a brief description of the ratings on "Understanding the task" criterion illustrate 
the general approach. The foiu* scale points for the holistic rating of the set of best pieces 
were: 

1. Totally misunderstood in more than half of the best pieces, 

2. May have understood or read beyond the surface problem in some instances 
but only partially understood more than half of the time or totally 
misunderstood the problem in two or more ir\stances. 

3. Understood or read beyond the surface problem most of the time but 
partially understood or misunderstood in some instances. 

4. Understood the task most of the time and read beyond the sxu'face problem at 
least a couple of times. 

Each of rhese descriptions was accompanied by as many as 10 combinations of 
scores from individual pieces that would produce the aggregate rating. Only the five 
best pieces were counted, regardless of whether the student included one or two more 
(as guidelines permitted). 

The mathematics portfolio assessment was accompanied in the 1991-92 statewide 
implementation by a sample-based administration of the State's new Uniform Test of 
mathematics. The Uniform Test was a matrix-sampled, mixed-format test, combining 
multiple-choice and open-ended items. Unlike the portfolio assessment, the Uniform 
Test was designed and scored by Advanced Systems for Measurement in Education, 
Vermont's testing contractor dxu'ing the 1991-92 school year. 



WRITING 

The design of the writing assessment, which was largely completed during the 
1990-91 pilot year, is substantially different from that of the mathematics assessment. In 
writing, students' portfolios must include a set number of pieces of specified types, the 
entire portfolio is rated, and a single best piece is chosen.^ 

In grade 4, each students' portfolio must irurlude: 

1. A table of contents. 

2. A single best piece, which is selected by the student, can come from any class 
and need not address an academic subject. 

3. A letter explaining the composition and selection of the best piece. 

4. A poem, short story, or personal narration. 

5. A personal response to a book, event, ciirrent issue, mathematical problem, 
or scientific phenomeiu>a 

6. A prose piece from any subject area other than English or language arts. 

The requirements for eighth grade are the same except that the portfolio must include 
three prose pieces. 

The best piece, the rest of the portfolio, aiKi performaru:e on the Uruform Test of 
writing (which is a direct writii^ task using standardized coiKlitiorxs and a single 
prompt) were all scored on the same five dimerxsions: 

• Purpose; 

• Organization; 

• Details; 

• Voice/Tone; and 

• Usage/ mechanics/grammar. 

A single four-point scale was used with all five dimensions, labeled as: 1. rarely; 
2, sometimes; 3. freqxaently; 4. extensively. The descriptions of the scale points, however, 
are generally phrased in terms of quality or exiensheness, not frequency. For example, in 
the case of purpose, the description of "sometimes" is: 

•*The following description is taken largely from "This is my Best:" Vermont's Writing 
Assessment Program, Pilot Year'^990'9^ (Vermont Department of Education, 1991b). 
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• "Attempts to establish a purpose. 

• Demonstrates some awareness of audience and task. 

• Exhibits rudimentary development of ideas" (Vermont Department of 
Education, 1991b, p. 6), 

In the 1991-92 statewide implementation, teachers scored their own students' 
portfolios and best pieces. Advanced Systems scored the Uniform Tests and arranged 
for a sample of portfolios to be drawn from each class for "moderation '-that is, to be 
scored by an external panel of teachers so that the scores of participating teachers co\ild 
be calibrated to a common standard. 

THE RAND/CRESST STUDIES 

The characteristics of the Vermont assessment program required that the 
RAND/CRESST evaluation be broad in scope. The RAND/CRESST evaluation is a 
series of interrelated efforts designed to gather information about: 

• The implementation and operation of the program at the school and 
classroom level; 

• The quality of measurement (including reliability and validity); and 

• Effects on instruction and on other aspects of schooling. 

These questions have been addressed with a variety of methods, including 
questionnaires admiiustered to teachers, interviews of teachers and principals, 
classroom observation, qualitative analysis of student portfolios, analysis of scoring 
methods and rubrics, questionnaires administered to scorers, and analysis of student- 
level and school-level scores. 

Our initial efforts focused primarily on the mathematics portfolio program. The 
mathematics program represented more of a qualitative break with past instructional 
practice than did the writing program. In addition, large-scale direct assessments of 
writing are quite common, even if large-scale portfolio assessments are not, and there is 
a research literature spanning decades about the characteristics of such assessments. In 
contrast, those building the Vermont mathematics assessment were largely plowing new 
ground, and their experiences could have important implications for the nationwide 
effort to develop direct assessments in miathematics. Accordingly, many of the results 



reported here pertain specifically to the mathematics program, while others pertain to 
both subjects. 

The RAND/CRESST evaluation is formative. Our expectation, like that of the 
State Department of Education, is that the program will require a long period of 
development. Our evaluation is designed to monitor that process and to provide 
frequent corrective feedback along the way. The results reported here, which are limited 
to the pilot year (1990-91) and the first year of statewide implementation (1991-92) reflect 
only the iiutial stages of the program's evolution. 

THE CONTENTS OF THIS REPORT 

Because of the state's need to use the results of the RAND/CRESST study for 
political dedsiorunaking and program design, the results of the 1990-91 and 1991-92 
studies have been released piecemeal. This report iiKorporates material released earlier 
and adds information not previously published. One interim report, released in the 
summer of 1991 (Koretz, Stecher, and Deibert, 1992, often labeled "RAND I" by 
educators in Vermont), provided iiutial findings on implementation and impact from 
our questionnaires and interviews with principals. Chapter 2 of this report summarizes 
those findings and adds information from our interviews of teachers and our qualitative 
analysis of portfolios. A second interim report, released in December 1992 (Koretz, 
McCaffrey, Klein, BeU, and Stecher, 1992, often called "RAND IF*) provided basic 
information on the reliability of scores. Chapters 3 and 4 of this report incorf)orate 
ir\formation from that report on the reliability of student scores and add additional 
detail, including a genera lizability-theory analysis. Chapter 5 of this report discusses 
implicatior\s of the student-level rater reliability for the reporting of aggregate scores. 
Chapter 6 describes preliminary analyses of validity. Chapter 7 discusses other 
implicatior\s of the data reported here. 
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2. IMPLEMENTATION AND IMPACT 



INTRODUCTION 

Our examiration of the implementation and impact of the Vermont portfolio 
assessment program began when the portfolios were introduced on a limited pilot basis 
in 1990-91 and continued during the 1991-92 school year, when portfolios were used by 
all fourth and eighth grade mathematics and writing teachers in most of the state s 
districts. We focused our investigation primarily on the mathematics portfolio program 
because it represented a clearer break from extant practice and was, from a national 
perspective, more unusual. Unless otherwise noted,^ the results reflect teachers' and 
principals' responses to the mathematics portfolios. 

Portfolios are seen both as evaluative tools and as levers to reform mathematics 
curriculum and instruction. In evaluating the implementation of the program, we tried 
to determine how well this vision was instilled through training and instantiated in 
classroom practices. 

The broad evaluation questions we sought to answer were: 

• How well was the portfolio assessment program implemented? 

• What effects did the portfolio assessment program have on schools and 
classrooms (ircluding effects on the content of the curriculum; the style and 
method of instructioa and the attitudes of teachers and students)? 

• What burdens did the portfolio assessment place on schools and classrooms? 

The results reported below represent only a snapshot of an innovation in its early 
stages; subsequent evaluation of the program in later years will reveal more about the its 
long-term effects. 

IMPLEMENTATION 

Data from the teacher questionnaires and the principal and teacher interviews 
permit us to examine three broad questions related to implementation: 



^Primarily in the discussions of implementation problems, teacher attitudes and the 
burdens imposed by the portfolios. 



• How weU-prepared were teachers to use the mathematics portfolios? 

• In what ways did teachers use the mathematics portfolios, and were portfolio 
practices similar across teachers? 

• What were the major implementation problems encountered by teachers and 
principals in implementing the portfolio program in both mathematics and 
writing? 

Teacher Prepaiation 

Vermont provided teachers with many opportunities to learn about the 
mathematics portfolio assessment program and prepare for its implementation. L>uring 
the pilot year (1990-91), the state sponsored a statewide orientation meeting in the fall 
and a series of regional workshops during the school year. In 1991-92, inservice training 
was expanded to include summer and fall mathematics institutes and preparation-for* 
scoring workshops during the year. ^ teachers were given a Resource Book with sample 
tasks and a Teachers' Guide with operational guidelines.^ In addition, regional rvetworks 
were established with consultants to provide supplemental training and support at the 
grassroots level. 

Survey and interview responses presented a somewhat inconsistent picKire of 
teachers' preparation to use the mathematics portfolios. Teachers ir4dicated they were 
satisfied with the preparation they received, but they also said they continued to 
encounter problems they were not prepared to address. Specifically, 

• Teachers generally said the state-sponsored mathematics portfolio training prepared 
them adecjuately. Virtually all respondent mathematics teachers attended one 
or more state-sponsored training sessions, and approximately three-quarters 
of the teachers said these sessions prepared them adequately to work with 
the portfolios. Similarly, three-quarters of those teachers who received 
supplemental network-level training thought it prepared them at least 
adequately to work with the portfolios. Most teachers used the Resource Book 
distributed by the state as a key source of mathematics portfolio tasks during 
the year.^ 

^Vermont Department of Education, Vermont Mathematics Portfolio Project Resource Book, 
Montpelier: author, September, 1991; and Vermont Department of Education, Vermont 
Mathematics Portfolio Project Teacher's Guide, Montpelier author, September, 199L 

^Changes in teachers' responses to the Resource Book illustrate the evolutionary impact of 
the program on instruction. During the first full year of implementation, many teachers had 
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• Nevertheless, ieadiers felt under-prepared to implement the mathematics portfolios 
at least some of the time. Approximately three-quarters of the teachers said 
they occasionally or frequently felt they did not have enough training to use 
the mathematics portfolios. Teachers commented that training left too miany 
unanswered questions, was not extensive enough, and did not address the 
problems of scoring. A few principals also felt that the training was 
inadequate; they noted particularly the lack of training in using the portfolio 
scoring criteria. 

Mathematics Portfolio Use 

The Vermont portfolio assessment was designed to "encourage \ocA 
inventiveness, [and] preserve local variations in curriculum and approach to teaching" 
(Mills and Brewer, 1988, pp. 3,5). The official guidelines for the portfolios deliberately 
leave much to the determination of local teachers. As a result, it is important to examine 
how teachers are actually implementing the portfolio assessment. Our siirveys and 
content analysis irnlicate substantial variation in implementation that could affect the 
validity of comparisons uased on the portfolio scores as well as their impact on student 
learning. Specifically, 

• There were important differences m the ways teachers used the mathematics 
portfolios m 1991-92, Teachers had different rules regarding authorship, 
revision, and the selection of tasks. Two-thirds of fourth grade teachers and 
44 percent of eighth grade teachers set ground rules for the amount or type of 
assistaiKe students could receive with portfolio pieces; the remainder of the 
teachers did not. Approximately one-half of the teachers said portfolio pieces 
were revised one time; about one-quarter of the teachers reported no 
revisions, and about one^uarter reported two or more revisions. 

A review of the contents of portfolios from three fovirth grade classes and three 
eighth grade classes found wide variation in the tasks include in portfolios within a 

difficulty finding tasks that were appropriate for portfolios, and the Resource Book was seen by 
many as a valuable source. By the end of the 1992-1993 school year, however, teachers' 
understanding of the types of problems that students should be given had evolved markedly, and 
many teachers had come to view some of the problems in the Resource Book as inadequate. 
Indeed, at the state scoring session in June, 1993, a number of teachers suggested "recalling' the 
Resource Book because of a perceived inadequacy of the tasks it includes. 



given class and even wider variation in portfolio tasks between classes. TTiere were 70 
different tasks included in the portfolios of the three fourth grade classrooms we 
examined, and there were 90 different tasks i:\ the poitfoUos of the three eighth grade 
classrooms.** Together, there were 154 different tasks listed on the tables of contents of 
the six classrooms (100 portfolios) we examined (see Tables 1 and 2). 

Within a classroom, many tasks appeared in only one student's portfolio, while 
only a few appeared in the majority of the portfolios. The average percentage of 
portfolios containing each task ranged from 13 percent to 50 percent, with most classes 
falling below 30 percent This indicates a moderate amount of variation in the selection 
of tasks within classrooms. Another nt\easure of within-class variation is the percentage 
of tasks that appear in one-half of the portfolios. Very few tasks met this benchmark in 
any of the classrooms, and in no classroom did we find a task that appeared in all of the 
portfolios (see Tables 1 and 2). 

There was even greater variation in task selection between classrooms. Only two 
tasks were used by more than one classroom in each grade. Perhaps most striking is the 
fact that more tasks were shared between grade levels (seven) than among classes within 
grade levels (four; see Table 3). 



^ Some tasks appear at both grade levels; therefore the overall count of tasks differs from 
the sum of the grade level counts. Similarly, grade level counts differ from the sum of the 
classroom counts because some tasks appear in more than one classroom. 
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V«rmont Math Portfolio Task Variation in Tliraa Fourth Orada Claaaroona 



Qrada 4i 43 portfolioa^ 70 diffarant taaka 



Claaaroon 4-1 

Number of portfolios 11 

Number of tasks per portfolio 6-7 

Number of different tasks in classroom 28 

Average number of times each task appeared 2.6 

Average percent of portfolios containing each task 23% 

Number of tasks appearing in more than 50% of the portfolios 3 
Claaaroom 4-2 

Number of portfolios 14 

Number of tasks per portfolio 5-7 

Number of tasks in classroom 26 

Average number of times each task appeared 3.C 

Average percent of portfolios containing each task 21% 

Number of tasks appearing in more tbiin 50% of the portfolios 0 
Claaaroom 4-3 

Number of portfolios 18 

Number of tasks per portfolio 7 

Number of tasks in classroom 18 

Average number of times task appeared 7.0 

Average percent of portfolios containing each task 39% 

Number of tasks appearing in more than 50% of the portfolios 4 
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Table 2 

Varmont Hath Portfolio rumk Variation in Three Eighth Grade Claeeroome 

Grade 8x 57 portfolioe, $0 different taeke 

Claeeroom 8-1 

Number of portfolios 12 

Number of tasks per portfolio 5-7 

Number of different tasks in classroom 34 

Average number of times each task appeared 2»4 
Average percent of portfolios containing each task 20% 

Number of tasks appearing in more than 50% of the portfolios 2 
Claeeroom B-2 

Number of portfolios 33 

Number of tasks per portfolio 5-7 

Number of tasks in classroom 47 

Average number of times each task appeared 4.2 
Average percent of portfolios containing each task 13% 

Number of tasks appearing in more than 50% of the portfolios 2 
Claeeroom 8-3 

Number of portfolios 12 

Number of tasks per portfolio 5-7 

Number of tasks in classroom 12 

Average number of times task appeared 6.0 
Average percent of portfolios containing each task 50% 

Number of tasks appearing in more than 50% of the portfolios 6 
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Tabla 3 

V#nont Math Portfolio Task Variation Fstwasn Classroou 



Number (%) of Tasks 
Number of in More Than One 
Sample Tasks classroom* 



Overall 




154 


8 


(5.2) 


Qrade 4 




70 


a 


(2.9) 


Classroom 


4/200 


28 


0 


(0.0) 


Classroom 


4/201 


26 


2 


(7.7) 


Classroom 


4/212 


18 


2 


(11.1) 


Qrada 8 




90 


2 


(a. 2) 


Classroom 


8/000 


34 


2 


(5.9) 


Classroom 


8/201 


47 


1 


(2.1) 


Classroom 


8/203 


12 


1 


(8.3) 



NOTE: * Percentages calculated based on nuinber in each row. 



Despite the great variation in the assignment of portfolio tasks and in rules 
governing their use, there were also important similarities in the ways teachers used the 
mathematics portfolios. For example, almost all teachers emphasized student interest 
and mathematical correctness as the key criteria for selecting pieces to go into the 
portfolios. Similarly, in both fourth and eighth grades, all students in participating 
classes compiled portfolios.^ 

Thes* iifferences in mathematics portfolio practices nuiy affect the interpretation 
of portfolio -based scores and therefore the validity of comparisons based on them. 
Some have suggested that school or district-level aggregate portfolio scores be used for 
accountability purposes, and Vermont has discussed using portfolio scores to compare 
schools or larger units.^ However, comparisons of outcomes based on dissimilar 



5 In addition, 80 percent of the eighth grade teachers used portfolios in only one or two 
classes during the 1991-92 school year. 

^Vennont \u^3 many schools with very small enrollments, and some districts contain only a 
single school at any given grade level. These small enrollments would undermine comparisons 
on any measure of student characteristics. Accordingly, a commonly discussed alternative would 
be to use portfolio scores to compare "Supervisory Unions," administrative units that typically 
include several schools and local districts. 



practices may lead to invalid inferences about differences in student performance. For 
example, it is reasonable to think that the number of times a portfolio piece is revised 
will affect its quality, which will, in turn, affect the scores it receives on the state's seven 
criteria. These differences could cause students with similar levels of proficiency to 
receive different scores and could obscure real differences in student proficiency. 
Similarly, the choice of tasks students are asked to perform and choose to include in 
their portfolios will determine the skills they demonstrate, which will affect the scores 
their portfolios receive.^ Such differences in task selection could also make portfolio 
scores a distorted indicator of students' act\ial performance. In addition, there is some 
evidence that other aspects of classroom context may affect students' performance on 
particular tasks. Dxiring the pilot year, some of the very same mathematics tasks were 
listed as "most successful" and "most tmsuccessful " by different teachers. 

• Variaiion m mathematics portfolio use was less marked in 1991-92 than in the 1990 
pilot year. During the pilot year there was much greater between-teacher 
variation in portfolio practkes than those noted above. Some eighth grade 
mathematics teachers selected a sir\gle class for participatbn, others used 
portfolios in all their classes. While the typical portfolio contained six pieces 
of work, the number of pieces ranged from zero to twenty. About one- 
quarter of the classes did not bother to select students' "best pieces" at all. In 
some classes the selection was left up to the student, in others it was made 
jointly by student and teachers, and in some cbsses teachers decided. The 
criteria used to choose best pieces were eqtially varied, ranging from pieces 
that made the students feel good to pieces that were likely to score well. 

During the first year of statewide implementation there was much more 
consistency in all these aspects of mathematics portfolio use. While there was still 
considerable variation in portfolio practkes, the state had taken steps to standardize 
many aspects of portfolio use that had been troublesome the year before Implementation 
Problems 

^This task selection problem has two components. First, some tasks may offer less 
opportunity than others to demonstrate a given skill. Second, some tasks may make it easier to 
demonstrate a high level of performance with respect to a given skill. Since the pilot year, some 
mathematics portfolio scoreis have stated that some teachers assign tasks that are too easy for 
their grade levels, apparently to increase shadents' ability to show the attributes scored. 
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A reform of this scale ar\d scope is bound to encounter some problems of 
implementation. In the case of the portfolios, the most serious problems were 
continuirxg confusion on the part of some teachers about the purpose of the portfolios 
and the procedures they were supposed to use, the perceived inadequacy of information 
from the state about the innovation, and the rapid speed of the reform. The latter 
problems occurred both in writing and mathematics. Specifically, 



• Teachers and students remained confused at times about the use of the mathematics 
portfolios. Three-quarters of the teachers were occasionally or frequently 
coT\fused about what they were supposed to do with portfolios or how they 
were supposed to do it. About one-half of the fourth grade teachers and one- 
third of the eighth grade teachers reported that students often vrere confused 
by portfolio activities, as well, 

• Many teachers and had difficulty finding appropriate mathematics tasks. Despite 
the dissemination of the Resource Book, more tha n one-half of the mathematics 
teachers reported at least occasional difficulty finding tasks appropriat » for 
inclusion in the portfolios. Moreover, discussions with raters during 
statewide scoring sessions in the spring of 1993 suggest that the true 
proportion may have been even higher, because they reported that some 
teachers included inappropriate materials (such as drill sheets) in the 1991-92 
portfolios. 

• Poor communication, insufficient information, and the rapid pace of implementation 
posed problems for many teachers in both umting and mathematics, Principab had 
the strongest opiruons about these issues. Approximately one-third of the 
principals felt the state administered the program poorly. They complained 
about unclear expectations, late or contradictory information, or poor 
communication from the state. Twenty percent of the principals also 
mentioned the need for more attention to scoring, and a few (10 percent) 
complained that the speed with which the assessment was implemented 
statewide was a source of stress. 

The responding mathematics teachers' comments supported these assertions. 
Approximately one-third of the teachers who were interviewed felt that their work 
could have been made much easier by a longer implementation period and wider 
availability of tasks. 

ERIC 
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The Vennont Department of Education has been responsive to past concerns of 
this sort For example, pilot year training was widely perceived to be insufficient. 
Teachers were dissatisfied with state-sponsored workshops because they failed to 
provide specific guidelines for implementing portfolios, adequate numbers of examples 
of appropriate activities, and clarification of the criteria to be used to judge the 
portfolios. Furthermore, once the state-sponsored regional workshops were completed, 
teachers received little, if any, additional support from colleagues or from state 
consultants. Many teachers felt isolated and under-prepared. 

The Department of Education tried to address these concerns in 1991-92 by 
broadening training, publishing specific guidebooks for teachers, arvi creating regional 
portfolio assessment networks. The Resource Book and the Teacher's Guide appeared to be 
quite helpful to teachers. In fact, as noted above, the Resource Book was teachers' chief 
source of portfolio tasks. The Network Consultants were another source of information, 
and those teachers who had contact with the consultants thought their feedback was 
helpful. It vw)uld appear that these efforts were improvements, but that teachers still 
had tuimet needs in the area of training. 

EFFECTS 

According to reports by teachers and principals, the portfolio assessment program 
was the impetus for diverse changes in mathematics ciuticulum and instruction, and it 
engendered both positive arxl negative reactions aoiong teachers and students. Data 
from the questionnaires and interviews provide infonrtation about the following 
questior\s: 

• What effects did the mathematics portfolios have on the content of the 
curriculum and the style of mathematics instruction? 

• In what ways did the mathematics portfolios affect student performance? 

• What effects did the portfolios (both mathematics arxi writing) have on 
teacher and student attitudes, including teachers' judgment of student 
abilities? 

Changes in Curriculum Content and Instructional Style 

The mathematics portfolio assessment had a substantial impact on curriculum 
content and it appeared to have some effects on instructior^l activities and style. 
Specifically, 
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• The amount of classroom time devoied to problem solving increased. More than 
three-quarters of the teachers reported spending more time teaching problem 
solving strategies than they had prior to the portfolios, and approximately 
one-half reported spending more time teaching patterns and relationships. 
Between one-third and one-half of the teachers sxirveyed reported spending 
less time on computatioa® 

There was little change in the amount of time spent on the topic of 
measurement/geometry (see Table 4). 

Tabl« 4 

Chu0« in cla«a TIm Devot«d to Mathnatical Topics 
(p«rc«nt«0e of teachers)* 





Somewhat 




Somewhat 




or Much 


About 


or Much 


Topic 


Less 


the Same 


More 




Grade 4 






Computation and algorithms 


49 


53 


7 


Estimation 


8 


55 


37 


Pat terns /Relationships 


5 


46 


48 


Measurement /Geometry 


15 


57 


27 


Problem Solving strategies 


2 


16 


82 


Orada 8 


Computation and algorithms 


31 


66 


3 


Estimation 


3 


63 


33 


Patterns/Relationships 


10 


40 


50 


Measurement /Geokue try 


3 


73 


23 


Problem Solving Strategies 


0 


23 


77 


♦Percentages may not sum to 


100 because of 


rounding 


errors , 



Similarly, most aspects of problem solving received more attention than they had 
prior to the math portfolios (see Table 5). Only single-step word problems did not 
receive mor^attention from the vast majority of teachers. Teacher interviews confirmed 

^Although not asked directly about areas of reduced emphasis, a small number 
(approximately 10 percenc) of the teachers interviewed volunteered that they spent less time on 
computation than previously. 
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these findings. Increased attention to problem solving and the use of new kinds 
problems were the most frequently mentioned changes in teaching. 



T«bU 5 

Change in TiiM on Probl«a Solving Activities 
(percentage of t€ach«r»}* 





Somewhat 




Somewhat 




or Much 


About 


or Much 


ToDic 


Less 


the Same 


More 


Grade 4 


Exploring mathematical 








patterns 


1 


38 


51 


Single-step word problems 


16 


64 


20 


Multiple-step word problems 


8 


38 


54 


Logic or reasoning problems 


1 


23 


76 


Applying math knowledge to 








new situations 


1 


26 


73 


Collecting, analyzing, 








report ina data 


1 


27 


71 


Orede 8 


Exploring mathematical 








patterns 


3 


41 


56 


Single-step word problems 


13 


69 


19 


Multiple-step word problems 


0 


41 


59 


Logic or reasoning problems 


0 


25 


75 


Applying math knowledge to 








new situations 


0 


19 


81 


Collecting, analyzing. 








reporting data 


0 


41 


59 


♦Percentages may not sum to 


100 because of 


rounding 


errors. 



• Teachers ' emphasis on mathematical communication increased. Approximately 
two-thirds of the teachers reported that they placed greater emphasis on oral 
discussions of mathematics, making charts and graphs, and writing reports 
than they had prior to the portfolios (see Table 6). Some of the teachers who 
were interviewed specifically mentioned spending more time on writing in 
the area of mathematics. 

• The amount of time students ivorked in small groups and in p^iirs increased. 
Approximately or«-half of the teachers reported that more mathematics 
work was done in small groups and in pairs than in prior years. More group 
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work was also mentioned during the teacher interviews. In comparison, 
most teachers reported no change in the amount of time students spent 
working individually or in whole-class activities. 

• Schools xx>luntarily expanded the use of portfolios to other grade levels. Nearly one- 
half of the principals indicated that the portfolio program had already been 
expanded beyond the fourth and eighth grades, and others said they 
intended to expand it in the future. Many teachers also reconunended 
expanding the program; they commented that for the program to be a success 
it must be implemented in all elementary and middle school grades. This 
occurred despite the considerable burder\s that accompanied the use of the 
portfolios (see below). 

• Other mathematics instructional practices appear to have changed to some degree. 
The majority of all principals interviewed stated that the program has had 
beneficial effects on matherruitics instruction, including in their responses a 
variety of char^ges in curriculum content as well as instructional methods or 
styles. The effects they mentioned were diverse, including irxrreased 
emphasis on 'flexible thmking," lessened reliance on textbooks, less emphasis 
on drill and practice, increased reliance on hands-on learrur^, increased use 
of interdisciplinary projects, and an increased emphasis on communication of 
mathenuitics. 

In response to another question, almost one^half of the principals interviewed 
mentioned the value of the portfolios as an educational intervention. They also reported 
positive changes in curriculum, better communication and collaboration among 
teachers, higher levels of thir\king and work, a broadening of individuals' views of 
mathematics and of matherruitics activities, a movenient away from traditional 
mathematics (by, among others, teachers who otherwise would not have made those 
changes), and a lesserung of "math phobia. ' 

Finally, one-quarter of the principals stated that it was too early to accurately 
assess the impact of the portfolio program on mathematics instrijction. 



42 

ERIC 



- 22 - 



Table 6 

Chati0« in Bmphaaia on Aapacta o£ Matbamatical Connunication 
(parcantaga oC taachara)* 





Somewhat 




Somewhat 




or Much 


About 


or Much 


Topic 


Less 


the Same 


More 


Orada 4 


Describing personal 








experiences 


2 


47 


51 


Oral discussions of 








mathematics 


0 


31 


69 


Making charts, graphs. 








diagrams, etc. 


3 


28 


68 


Written reports about 








mathematics 


1 


33 


66 


Orada 8 


Describing personal 








experiences 


3 


69 


28 


Oral discussions of 








mathematics 


6 


44 


50 


Making charts, graphs. 








diagrams, etc. 


0 


14 


66 


Written reports about 








mathematics 


3 


25 


75 


♦Percentages may not sum 


to 100 because of 


rounding 


errors . 



Changes in Student Mathematics Performance 

We did not attempt to assess changes student performance directly, but teachers 
were asked to coirunent on char^ges they observed in the work students produced, 
particularly diffenences between portfolio tasks and traditional mathematics 
assignments* Teachers reported considerable change in student performance, which 
affected their own opinions about student ability. Specifically, 

• Regardelss of ability level, most students performed differently on portfolio tasks than 
on regular mathematics assignments. Only about one-third of teachers said 
students performance was the same on both types of tasks. About one- 
quarter of the teachers said performance varied so greatly across tasks or 
across students that they could not make overall comparisons. Of the 
remaining teachers, most reported that students did more poorly on portfolio 
tasks than traditional assignments. 
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Similarly mixed results were reported when teachers were asked to focus on low- 
ability students. Approximately one-quarter of the teachers reported that low-ability 
students were frequently more successful as a result of the portfolios, and they were 
"occasionally" more successful in another 40 percent of the classrooms (see Table 8). On 
the other hand, virtually all fourth grade teachers and 80 percent of the eighth grade 
teachers indicated that low ability students had difficulty with portfolio tasks at least 
occasionally (see Table 7). 

• Teachers' judgments of students ' mathematics abilities changed as a result of student 
portfolio work. More than 80 percent of teachers said they had changed their 
opinion of students' mathematical ability on the basis of students' portfolio 
work. Although the amount of change reported by most teachers was small, 
the pervasiveness of change was striking. Moreover, or^third of teachers 
said they changed their opiruon of stijdents' abilities a ''moderate amount," 
and nearly ten percent changed their opinions "a great deal.'* Pilot year 
teachers said the portfolios permitted students to demonstrate greater 
creativity and differentiate themselves one from another more than 
traditional assignments. 

Changes in Teacher and Shident Attitudes 

The introduction of the portfolios caused changes in teacher and student attitudes 
towards curriciilum, instructioa and learning. On the positive side, there was inaeased 
enthusiasm among mathematics teachers for their subject; on the negative side, both 
mathematics and writing teachers were frustrated about changes that were often 
perceived to be demanding and sometimes perceived as threatening. Specifically, 
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Tabltt 7 



Frequency of Port £olio-R« let ed Problw 
(percentage ot teachers reporting)* 



Rarely or 


Occasion- 


Often or 


I ssue 


Neve r 


ally 


Always 


Orade 


4 






I don't understand what I*m 








expected to do 


24 




z o 


I don t nave enougn crainin^ 








now CO uo ic 




39 


36 


I nave Qiiiicuity iinain^ 








appropriate tasks 




42 


33 


I lacK c ime to prepare poruLoiio 










15 


25 


59 


Noc enou^n Lxme uo cover une Luj.i. 










4 


15 


81 


LtOW aoxxxuy auuueiiua ncivc 








QlLLlCUXuy Wlun UaSIvS 


3 


37 


60 












9 


45 


45 


SCUQencs oon t isnow now uo ooxve 








oT*o^^1 omc 


7 


43 


50 


SCUQencs noc inueresk.eQ xii 








nortfolio tasks 


29 


50 


21 


ane 








I don*t understand what I'm 








cA^c^UcU U O UO 


27 


^7 


16 


I don't have enough training in 








how to do it 


38 


43 


19 


I have difficulty finding 








appropriate tasks 


41 


41 


19 


I lack time to prepare portfolio 








lessons 


14 


22 


65 


Not enough time to cover the full 








math curriculum 


U 


24 


65 


Low ability students have 








difficulty with tasks 


19 


31 


50 


Students don*t understand what to 








do with tasks 


11 


50 


39 


students don't know how to solve 








problems 


8 


57 


35 


Students not interested in 








portfolio tasks 


16 


51 


32 


•Totals may not sum to 100 percent 


due to 


rounding errors 





Portfolios increased mathematics teachers' enthusiasm for their subject and had 
other pontwe effects on teadiers ' attitudes. Surveyed mathematics teachers 
reported that the portfolio program had a number of positive effects on their 
own feelings about teaching mathematics. For example, more than one-half 
of the teachers said they were frequently more enthusiastic about teaching 
math, and over 90 percent were more enthusiastic at least occasionally. 
Similarly, over 40 percent said the goals of mathematics instruction were 
improved and math was mora closely linked to other subjects (see Table 8). 

Tabl« 8 

Prequeney of Positivs Effects 
(peresntsg* of teachers reporting)* 





Rarely 


Occasion- 


Often 


Issue 


or 


Never 


ally 


or Always 


Orede 


4 








I am more enthusiastic about 










teaching math 




15 


29 


56 


Goals of math instruction are 










improved 




10 


33 


57 


Math is more closely linked to 










other subjects 




14 


41 


45 


Students* attitudes toward math 










improve 




19 


36 


43 


Students are learning more 










mathematics 




14 


35 


51 


Low ability students are more 










successful 




28 


40 


33 


Orede 


8 








I am more enthusiastic about 










teaching math 




11 


38 


51 


Goals of math instruction are 










improved 




14 


57 


30 


Math is more closely linked to 










other subjects 




11 


46 


43 


Students* attitudes Coward math 










improve 




32 


41 


27 


Students are learning more 










mathematics 




14 


51 


35 


Low ability students are more 










successful 




38 


46 


16 


♦Totals may not sum to 100 percent 


due to 


rounding errors 
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Principals spoke more generally about the effects of the portfolios (both writing 
and mathematics) on teacher's attitiides. Although the great majority of principals 
characterized the attitudes of their teachers to the portfolio program as mixed, twenty- 
three percent mentioned specific positive feelings on the part of teachers. Only four 
percent of the principals characterized their teachers attitudes about the program as 
negative. About one-fifth of the principals (18 percent) made generic comments about 
teachers* positive responses to the concept of the portfolio program. 

• Teachers and principals perceived the portfolio program to be a substantial burden. 
Eighty-six percent of principals interviewed labeled the portfolio program as 
"burdensome" (without regard to subject matter), although they appeared to 
view the burdens as primarily resting on teachers rather than on themselves. 
(Burdens will be discussed in greater detail in the next section.) For example, 
twenty-six percent of the principals noted that teachers fotmd the time they 
had to spend on the portfolios to be burdensome* Ten percent mentioned 
that teachers felt they had to be out of the class too often in order to attend 
the necessary training sessions. Ten percent also stated that teachers felt it 
was inappropriate to have the 4th grade teachers burdened by both 
mathematics and writing portfolio programs. Some of these time pressures 
probably will decrease as teachers become more familiar with the irmovation; 
others are likely to persist. 

• Scoring xvas a particular source of anxieixf among teachers. During the pilot year, 
some teachers expressed concerns about the use of portfolios to judge 
students and school performance. These concerns continued in 1991-92. 
Roughly one-third of the fourth grade mathematics teachers and one-half of 
the eighth grade mathematics teachers voiced concerns about the validity 
and fairness of the scoring system and about the public reporting of scores. 
This is a significant response given that these were unprompted comments. 
Principals echoed these concerns more generally. About one-fifth of the 
principals mentioned problems their mathematics and writing teachers were 
having with scoring, including concerns about subjectivity, inconsistency, 
and trairung. 

• Students had mixed reactions to the mathematics portfolios. We obtained no direct 
measures of students' reactions to the portfolio program, but we did question 
teachers and principals about them. Over 40 percent of the fourth grade 
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mathematics teachers and about 30 percent of the eighth grade mathematics 
teachers said that students' attitudes towards math improved as a result of 
the portfolios and that students were learning more mathematics (see Table 
8), On the other hand, one-half of the teachers at each grade level reported 
that students strongly disliked writing about mathematics and explaining the 
thinking that went into their work, both of which were common elements of 
many portfolio tasks. 

Students' positive reactior\s to the mathematics portfolios included pride in their 
work, excitement over new challenges, interest in problems and problem solving, 
enjoyment of cooperative group work, fun with new types of activities, and enthusiasm 
for expressing themselves in writing. Students' rvegative reactior\s to the mathematics 
portfolios primarily centered on their dislike of writing, but also included frustration 
with new and unfamiliar activities, and, among eighth grade students, lack of 
understanding of purpose, anger over the amoimt of time involved, and occasional 
concern about the effect on their grades. 

Seventy percent of the principals described students' responses to the mathenwtics 
portfolios, and the positive responses far outweighed the negative ones. About one-fifth 
of the principals noted specific aspects of the mathematics portfolios about which 
students felt positive, including the types of problems they were working on, finding 
creative solutions, doing hands-on projects, using marupulative aids, using language 
arts in mathenutics activities, and doing interdisciplinary work. Some principals 
commented that portfolio work engendered feelii>gs of pride or ownership. Principals' 
comments about negative student reactions to the mathematics portfolios were diverse 
and difficult to summarize. Only one point was made by more than a single principal: 
Eight percent of the prirKiipals mentioned that students do rK>t like the portfolios 
because of the writir^g involved; in articular, they reported, students found it difficiilt 
to write down their thought processes and disliked doing so. About 10 percent of the 
principals observed that student attitudes toward the portfolios were related to teacher 
attitudes. 

BURDENS 

Change has come at a price. The portfolio assessment placed substantial new 
demands on students, teachers arKl principals. The surveys and interviews provide 
partial answers to two general questior\s: 
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• What demands did the mathematics portfolios make on teachers' and 
students' time? 

• What demands did the mathematics and writing portfolios make on other 
school resources? 

Time Demands 

The mathematics portfolios placed major time demands on teachers. 

• The mathematics portfolios required a significant amount of class time, which had to 
be taken from other activities. Eighty percent of the fourth grade teachers and 
65% of the eighth grade teachers had difficulty finding time to cover the 
regular mathematics curriculum. On average, fourth grade classes spent 
fifteen hours each month on the math portfolios, and eighth grade classes 
spent ten hours each month (see Table 9). Principals were aware of these 
efforts, and a large number of principals complained about the burdens of 
record keeping, logistics, and the demands on class time. 



Cl«s«rooDi TiM Sp«at on Portfolio Activitiss 
(hours p«x aontb) 



Activity 



Grade 4 Grade 8 Overall 

Mean (SO) Mean (SD) Mean (SD) 



Doing portfolio tasks for 
the first time 



7,8 (5.8) 5.3 (4.7) 7.1 (5.8) 



Revising or rewriting 
portfolio tasks 



4.1 (5.7) 2.2 (3.4) 3.6 (5.2) 



organi? ing/managing 
port f ol ioG 



3.0 (3.3) 2.3 (2.9) 2.8 (3.2) 



Total classroom time 



15.0 (13.Q) 9.9 (8.9) 13.7 (12.2) 
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One consequence of the increased time devoted to mathematics portfolios was a 
reduction in attention to other parts of the curriculum. Over 80 percent of fourth grade 
teachers and over 60 percent of eighth grade teachers reported that they often had 
difficulty covering the required curriculum (see Table 7). In interviews, several fourth 
teachers, who usually teach in self-contained classrooms, reported that the portfolios 
had forced them to spend more time on math and less time on other subjects. Several 
eighth grade teachers, who usually teach in departmentalized settings, reported they 
had to cover fewer math topics.^ 

These demands on classroom time are not likely to diminish significantly in the 
future; they are an essential part of the portfolios. If the portfolio assessment program is 
to be successful, teachers will need to find ways to acconunodate these demands on class 
time while mi sac^ iicing essential elements of the curriculum. 

• Mathematics portfolios required a significant amount of teacher preparation time. 
The portfolio program also demanded a large amount of time outside of the 
classroom. On average, teachers spent seventeen hours each month finding 
portfolio tasks, preparing portfolio lessons, and evaluating the contents of 
portfolios (see Table 10). Sixty percent of the teachers at both grade levels 
said they often lacked time to prepare portfolio lessons (see Table 7). 

Principals also noted that teachers had to fir\d time to review and manage the 
portfolios. Fourth grade teachers were especially pressed for time because they had to 
implement both writing and mathematics portfolios in the same year. 

In theory, preparation time should diminish over the next few years as teachers 
build a repertoire of appropriate tasks and activities that can be reused or adapted. 
There may also be a decrease in the time teachers spend managing the portfolios, as they 
develop strategies for handling these respor\sibilities more efficiently. It is unlikely, 
however, that these demands will ever disappear. 



^^is is consistent with teachere' responses to other types of high-stakes testing programs. 
(See, for example. Salmon-Cox, 1982. 1984.) 
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T«bl« 10 



M«ch«r TiflM Spmat on Portfolio Xetivities 
(hours p«r month) 



Grade 4 Grade 8 Overall 



Activity 


Mean 


(SD) 


Mean 


(SD) 


Mean 


(SD) 


Finding appropriate tasks 
and/or materials 


6.0 


(6.7) 


5.9 


(5.2) 


5.9 


(6.3) 


Preparing portfolio lessons 


6.3 


(5.1) 


6.9 


(6,8) 


6.5 


(5.6) 


Conducting portfolio lessons 


8.8 


(5.5) 


10.3 


(14.5) 


9.2 


(8.9) 


Helping students organize/ 
manage their portfolios 


3.7 


(4.5) 


2.7 


(2.8) 


3 .4 


(4.1) 


Scoring/evaluating the 
contents of portfolios 


4.6 


(7.1) 


6.7 


(7.6) 


5.2 


(7.31 


Total teacher time 


28.9 


(22.0) 


33.2 


(27.2) 


30.1 


(23. 6) 



Resource Demands 

Both the nuithenuitics and writing portfolios placed demands on school resources. 
Specifically. 

• Most principals provided releobe time to teachers for portfolio training and other 
activities. Almost all principals (92 percent) provided special support to their 
mathematics and writing teachers participating in the portfolio project, most 
often released time. About three-quarters of principals provided release time 
for teachers to attend state-sponsored trainin; workshops, and more than 
one-fourth of principals provided release time for teachers to work on 
preparing lessons, selecting best pieces, organizing filial portfolios and other 
portfolio activities outside of class during the school day. A few principals 
commented that the time required for training was excessive, and some 
teachers felt they were being asked to spend too much time away from their 
classes to attend portfolio traLning sessions. 
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The provision of release time was a substantial financial burden for participating 
schools, because it was common for schools to bear the cost of substitute teachers when 
release time was granted. Nonetheless, very few principals (13 percent) reported that 
they were unable to provide the support teachers requested. 

The need for '»xtensive training should decline as teachers become familiar with 
the basic elements of the portfolio assessment program. Additional state-sponsored 
traiiung continued during the 1992-93 school year. Beyond that, there will always be a 
need for some additional traiiung to prepare new teachers, to supplemental teacher 
expertise in new ways as the assessment reform matures, and to maintain standards in 
the implementation of the program and scoring of the mathematics and writing 
portfolios. However, it is likely that training demands will decrease somewhat over the 
coming years. 

CONCLUSIONS 

During 1990-91 and 1991-92, Vermont made important strides toward realizmg 
the goals of the portfolio assessment program for the reform of assessment and 
instruction. The innovation was widely adopted, and mathematics teachers reported 
paying substantially more attention to problem solvit^ and mathematical 
communication. There was also some evidence of changes in instructional practices, and 
the state appeared to be promoting greater acceptance and more effective use through 
impmved teacher training. There was widespread support for the reform at the school 
level; nearly one-half of the schools were voluntarily expanding the use of portfolios to 
other grade levels. 

However, substantial problems remain. From the practitior\er*s perspective, the 
mathematics portfolio assessment has created new burdens for principals, teachers and 
students. These burdens include demands on teachers' time and school resources for 
traiiung, preparation and use of portfolios. While some of the demands created by 
portfolios are likely to decline with experience, others represent continuing burdens. 

The variability we found in the implementation of the mathematics portfolio 
program, while consistent with the notion of a decentralized, bottom-up reform, was so 
substantial that it threatens both the impact of the program on instruction and the 
validity of the resulting data. In particular, we are concerned about the apparent lack of 
a common understandir^ of the fundamental constructs of problem solving and 
mathematical commuiucation. If the assessment program is to achieve its goals, the 
state will need to find ways to instill a shared understanding of these core constructs. In 
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addition, Vennonl needs to be concerned about teachers' idiosyncratic portfolio 
practices, which will affect its ability to interpret portfolio scores. 

These observations pertain only to the initial stages of this complex intervention; it 
is too soon to tell what the ultimate impact of the portfolios will be on curriculum and 
instruction or how useful they will be for classroom assessment. The answer two these 
questions will require more time for the program to mature and further study. 
Similarly, questions about teachers' understanding of problem solving and mathematical 
communication, the impact of these reforms on students from different backgrounds 
and students of different abilities, and the long-term acceptance and transformation of 
the vision of portfolios guiding Vermont's efforts will be answered only in the future. 
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3. RELIABILITY OF WRITING PORTFOLIO SCORES 



This chapter examines the reliability of scores assigned to writing portfolios. We 
found that writing scores were unreliable, because readers were inconsistent in their 
evaluations of the quality of students' work. (Chapter 4 examines the reliability ol scores 
assigned to mathematics portfolios, and Chapter 5 reports on the reliability of school- 
level portfolio scores.) 

"Reliability has many mearungs, however, so it is important to clarify which 
aspects of reliability we assessed. For the most part, we examined "rater reliability," 
which is the degree to which different raters agreed on the scores that should be 
assigned to portfolios or to iiKlividual pieces within them Rater reliability is critical, 
because it limits the amount of confidervce that can be pbced in the scores. However, it 
is only one aspect of the larger question of the reliability (or consistency) of scores, 
which is often called "score reliability.*' For scores to be reliable in the broader sense, 
they must be consistent, not only across raters, but also across different instances of 
measurement. For example, a score is not a reliable measure of a student's writing if it is 
specific to the particular essays the student is asked to write for the test. Comparison to 
a multiple-choice test clarifies this distinction The "rater reliability" of a multiple-choice 
test is perfect; the scanning machine and software will generate exactly the same score 
from a given answer sheet time after time. When publishers of multiple -choice tests 
report less than perfect reliability, they are referring to differences in students' scores on 
different forms of the test or on halves of the same test. 

High rater reliability is also necessary but not sufficient to indicate that scores are 
an accurate reflection of the quality of work on the specific tasks students have included 
in their portfolios. If two readers do agree on the scores that should be assigned to a 
portfolio, it could be because they both made accurate assessments of that portfolio's 
quality. However, a high degree of agreement between readers also could occur for 
reasons that have little or nothing to do with portfolio quality, for example, if they gave 
high marks to portfolios that were written neatly and low marks to messy ones. Low 
agreement rates necessarily indicate that at least one of the readers did not make an 
accurate assessment of portfolio quality. 

This chapter examines the following specific questions about the reliability of 
ratings of the 1992 Venrtont writing portfolios: 
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• Did readers agree with each other regarding the relative quality of the 
portfolios? 

• Was there more agreement among readers on some grading dimensions 
(often called "scoring criteria" in Vermont) than on other dunensior\s? Is 
there more agreement between readers regarding what scores should 
assigned to the best piece in a portfolio than there is on what scores shoxild be 
given to the combination of other pieces in that portfolio? 

• Were some readers generally more leruent than others, and does a student s 
own teacher generally assign a higher or lower grade to a portfolio than an 
independent reader? 

• Did students who received relatively high scores on one dimension also tend 
to receive high scores on other dimerxsiorxs? 

• Did students who received a high score on their "best" piece also tend to 
receive a high score on the remainder of their portfolio^ i*e., is a student's 
performance corxsistent aooss different parts of the portfolio? 

• To what degree would the reliability of portfolio scores be improved by 
increasing the number of independent readers who evaluate a portfolio or 
increasing the number of separately graded parts? 

• Are the answers to the questions above the same for Grade 4 as they are for 
Grade 8? For instance, were Grade 4 portfolios scored any more or less 
cor\sistently than Grade 8 portfolios? 

• How did the degree of agreement between readers in grading portfolios 
compare to the agreement rate in grading essays when all students wrote 
responses to the same prompt undef standardized conditions? 

CHAPTER OVERVIEW 

The r«xt portion of this chapter describes the procedures that were used to gather 
information about the degree of agreement between readers. We then show the typical 
agreement rate on a dimer\sioa This is followed by a summary of agreement rates on all 
dimensions at both grade levels, an analysis of some of the factors that may have 
contributed to these rates, a discussion of the effects of different strategies for improving 
the reliability of portfolio scores, and a comparison of reader agreement rates on 
portfolios with those on a uniform writing test. Our conclusions about reader 
agreement appear at the end of the cliapter. 
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PROCEDURES 

As noted previously, each writing portfoliiD had two parts. One part was the piece 
the student and /or teacher selected as the "best" one. The other part was the remaining 
5 to 7 pieces taken together as a set. These two parts are referred to as the "best" and 
"rest/* respectively. 

Both parts were graded on five dimensions by the classroom teacher: purpose, 
organization, details, voice and tone, and the combination of usage, grammar, and 
mechanics (see Appendix D for a description of each dimension). Thus, all told, each 
reader assigned 10 scores to a portfolio (5 to the best piece and 5 to the rest). Each of the 
10 scores was assigned on a ^hpoint scale. A reader assigned all 10 scores before goir\g 
on to the next portfolio. 

A sample of 1,903 Grade 4 portfolios and 750 Grade 8 portfolios were selected 
randomly from among those that were already scored by the student's own classroom 
teacher Each selected portfolio was graded again by oxve of the teachers who 
participated at a centralized jjortfolio grading workshop. This second reader did not see 
the scores assigned by the first reader, i.e., the student's own teacher. All of the second 
readers were drawn from the pool of Vermont classroom teachers, but none of them 
regraded portfolios from their own classroom. 

AGREEMENT BETWEEN READERS 

We addressed two primary questions about the degree of agreement between the 
scores assigned to portfolios by the student's own classroom teacher and those assigned 
by an independent reader. First, were the scores assigned by classroom teachers biased, 
i.e., systematically higher or lower? Second, to what degree were raters consistent with 
each other in deciding which portfolios warranted high and low scores? 

Bias 

Some observers expressed concern that students' own teachers might be more 
lenient in assigning scores. That did not hapf>en. On average, a student's teacher was 
oniy trivially more lenient than was the second rater. Averaged across dimensions, the 
scores provided by the students' own teachers were never more than 0.04 points (on the 
4-point scale) higher than those assigned by the second raters (Table 11). This striking 
similarity of average ratings was also quite consistent across specific scoring dimensions. 
In only four of the 20 comparisons (2 grades x 2 parts x 5 dimensions) was the mean 
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difference between raters greater than 0.05, and in no case did it exceed 0.10 (see 
Appendix A). 



T«bl« 11 

Ummn 8cor«s Acro«« DiMMion« By 
R«ad«r Typ«, Gr«d«, And Part 





Grade 


4 


Grade 


8 


Reader Type 


Best 


Rest 


Best 


Rest 


Classroom Teacher 
Independent Reader 


2.86 
2 .82 


2.70 
2.66 


2.99 
2.96 


2.77 
2.77 


Difference 


0.04 


0.04 


0.03 


0.00 



NOTE: Despite the large number of cases, these 
differences are not statistically significant. The 
t-values for best and rest at Grade 4 are 1.47 and 



1.30, respectively. The corresponding values at Grade 
8 are 0 .90 and 0 .02 . 

Consistency of Scores 

Even though the two raters assigned very similar average scores, the reliability of 
ratings was low: raters were not consistent in assigning scores to individual portfolios.^ 
That is, although they agreed on the average score across all portfolios, raters often 
disagreed about which portfolios warranted high or low scores. The small differences in 
mean scores noted above contributed virtually nothing to this inconsistency in ratings. 

This section discusses the evidence for this conclusion. We begin by illustrating 
the serbusness of the reliability problem by showing the degree of agreement on one 
scoring dimension for one part of the portfolio. We then present summary data to show 
that the very low level of agreement between readers in this slice of the data is typical of 
that found on both parts of a portfolio, on all five dimensions, and at both grade levels. 

We use two different approaches to summarize the reliability of ratings. One 
approach examines the proportion of cases in which both raters agree on a score. This 
percentage agreement must be interpreted cautiously, however, because some amount 
of agreement would be expected by chance alone. (The agreement expected by chance is 

^More precisely, they were inconsistent in assigning scores to the two parts of the 
portfolio, the best piece and the rest. Although we combined scores across the two parts and 
discuss the reliability of the combined scores later in this chapter, the best-piece and rest scores 
were not combined into an overall portfolio score by the assessment program. 
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what one would obtain if portfolios were scrambled randomly, so that one would be 
comparing the first rater s score for one student with the second rater's score for a 
randomly chosen student, who would in almost all cases be a different student.)^ The 
rate of agreement expected by chance varies depending on the number of score 
categories and the degree to which scores are concentrated in one or a few categories. 
When scores are highly coiKentrated at a single value, the rate of agreement expected by 
chance can approach 100 percent. (This happened in mathematics but not in writing; see 
Chapter 4.) A second conventional measiare of rater reiaibility is the correlation 
coefficient between raters. The correlation coefficient ranges from a value of 0.00 (if 
there is no relationship between the scores provided by two raters) to 1.00 (if there is a 
perfect relationship)^. 

The Typical Pattern 

Regardless of which index was used (percentage agreement or correlation 
coefficient), the degree of agreement between readers was low. It was generally similar 
on all five dimensions on both parts and was only slightly higher at Grade 8 than at 
Grade 4. 

Table 12 shows the extent to which different readers assigned the same or 
different scores to the best piece on the detail dimension at Grade 4. The rows 
correspond to the possible scores assigned by the first reader (i.e., the student s own 
classroom teacher). The columns refer to the possible scores assigned by the 
independent reader at the special gradir\g session. The entries in the table are the 
percentages of twice-scored portfolios receiving each of the 16 possible combinations of 

chance rate is a function of each reader's distribution of grades across the four score 
levels. If both raters assigned 25 percent of the portfolios to each of the four possible scores-a 
highly unlikely case-then one would obtain agreement in 25 percent of the cases by chance alone. 
More generally, however, the chance rake of agreement depends on the "maiginal percentages '- 
i.e., the percentages of portfolios assigned to each score. The chance rate is the sum of the 
products of the marginal percentages for each rater. For example, suppose that 33 percent of first 
raters and 32 percent of second raters assigned a value of I. The product of those percentages is 
.33 X .32 s .11 . Products are similarly calculated for scores of 2, 3, and 4, and the four products are 
summed, and that sum is the percentage agreement expected by chance. 

''We used the Spearman rank order correlation (rho) rather than the more common 
Pearson correlation because the difference between a 1 or 2 on the 4"point grading scale was not 
always viewed the same as the difference between a 2 and 3 or between a 3 and 4. In practice, 
however, the choice between the Spearman and Pearson coefficients made little difference. We 
did not use the Kappa statistic to quantify agreement rates because it is not sensitive to distances 
off the diagonal. Kappa treats a 1-point difference between readers the same as a 3-point 
difference between them. 



scores. For example, 1 percent of the portfolios received a score of 1 from both raters. 
However, in 2 percent of the portfolios, the first reader assigned a score of 1 and the 
second reader assigned a 2. On another 1 percent of the portfolios^ the first reader 
assigned a score of 1 and the second reader assigned a score of 3. 



Tabl« 12 

Ritt«r Aa^««M&t^ -Typical P«tt«rn 
(Beat Pi«G«*Mt«ils-Orad« 4 Htiti&g) 







Second 


Rater 






First 












Rater 


1 


2 


3 


4 


Total 


1 


1 


2 


1 


0 


4 


2 


2 


1« 


12 


3 


33 


3 


2 


12 


21 


9 


44 


4 


0 


3 


10 


7 


20 


Total 


s 


33 


44 


19 


100% 



Percentage of pieces on which the two readers agreed on what 
score to assign =1+16+2147= 45% 
Percent by chance alone > 36% 
Kappa = 0.17 
Spearman Rho • 0.35 

If the first and second reader in Table 11 agreed perfectly with each other on how 
each portfolio should be scored, then the values along the diagonal (from upper left to 
lower right) would sum to 100 percent. Instead, raters agreed with each other on only 45 
percent of the portfolios. (The sum of the diagonal entries is 1 + 16 *h 21 *h 9 » 45.) By 
chance alone, two readers would agree on the score assigned to roughly 35 percent of 
the portfolios on this dimer\sioa Thus, the observed degree of agreement, 45 percent, is 
only about 10 percentage points better than chance.^ 

The correlation coefficients between the grades assigned by the classroom teacher 
and the independent reader also showed that rater reliability was generally low (Table 
13). The correlations varied from a low of .28 (Voice/Tone for the "rest" in fourth grade) 
to .57 (Usage, Grammar, and Mechanics for the rest in eighth grade). Only one of the 20 
correlations exceeded .50, however, and the mean correlations (averaged across 



^me participants in the program maintiined that adjacent scores (e.g., a 2 and a 3) 
should count as agreement. We rejected this argument because the program uses only a four- 
point scale. With only four points on the scale, counting adjacent scores as agreement would 
mean that most scores count as agreement. For example, in the case of Table 12, ii the fiist rater 
gave a piece a score of 3 (the modal score), then any score assigned by the second reader other 
than ihe very rare value of 1 would count as agreement. 
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dimensions) were only .35 in fourth grade and .43 in eighth grade. These are low by any 
standard. A correlation of .35 means that one can predict about only 12 percent in the 
variance of second raters' scores by knowing the first raters scores; a correlation of .43 
means that one can predict about 18 percent. The remainder of the variance is enor. 



TabU 13 

Znt«r-Kat«r Corr«Xations« Writing 







Best 




Rest 


Dimension 


Grade 


4 Grade 8 


Grade 


4 Grade 6 


Purpose 


.33 


.34 


.33 


.39 


Organization 


.36 


.41 


.31 


.43 


Details 


.35 


.44 


.33 


.41 


Voice/ Tone 


.33 


.45 


.28 


.37 


U/G/M 


.40 


.46 


.43 


.57 


Mean 


.35 


.42 


.34 


.43 



RELATIONSHIPS BETWEEN SCORES ON THE TWO PARTS 

The Vermont assessment program was designed to provide two separate sets of 
scores for each portfoUo, one for the best piece and another for the rest. By design, no 
composite score across the two parts was created. Our understanding is that this design 
had a number of rationales. Keeping the two parts separate might be desirable in terms 
of incentives to students and teachers; that is, they might do different types of work in 
preparing the two parts. In addition the two parts were supposed to provide different 
views of a stvident's performance. 

To what degree were the parts of the portfolio actually independent of each other? 
We cannot ascertain whether the two parts provided different incentives in the 
classroom. However, we can assess the extent to which the two parts functioned 
independently as measurement tools, In this section, we compare mean scores on the 
two parts and examine the conelalions between them. In a later section, we use a 
technique called generalizability analysis to revisit this question. If the mean scores on 
the two parts are very similar and if the correlations between scores on the two parts are 
high, then having both parts contributes very little additional, independent information. 
In other words, in terms of measurement, including the second part does little to 
improve the information yielded by the assessment. 

Overall, our analyses showed that scores on the two parts were strongly related 
and that the inclusion of a second part added Uttle additional independent informatioa 
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This indicates that as of 1991-92, the program was largely unsuccessful in meeting its 
goal of using the two parts to garner different information about student performance. 

Mean Scorss 

The mean score on the best piece was usually only slightly higher than the mean 
on the rest (Table 11). This pattern held at both grade levels and for aU of the scoring 
dimensions. Because the best piece is by definition supposed to be better than others, 
these small differences between parts may indicate that the system is not functioning 
propt -ly. One possibiUty is that the students are not successfully identifying a best 
piece; that is, their best pieces are not much better than any of their other pieces. A 
second possibUity is that the rating system is not sensitive enough to discern differences 
in quaUty between best pieces and others. Yet another possibility is that raters have 
relative standards-that is, they implicitly grade best pieces harder than others. This 
would undermine the state's effort to interpret scores in terms of the stated definitions of 
each scale point, which are the same for both parts. 

Girrelations Between Parts 

Some of the simple correlations between scores on the two parts are low. 
However, this appears to be the result of rater unreliability. When the correlations 
between parts are adjusted for rater reliabiUty, they are quite Wgh, suggesting that there 
is relaHvely Uttle independent infomution added by including both parts. 

Averagpd across dimensions, the simple correlations between raters across parts 
range from .25 to .37 (the "raw " correlations in Table 14). The correlations across parts 
are somewhat higher in the eighth grade than in the fourth grade. Moreover, it makes 
Uttle difference whether scores are restricted to a single dimension, In fourth grade, for 
example, if one compares the score on one part to the score on the other, both on the 
same dimensiorv the correlation between the scores is .28. If one compares the score on 
one part and one dimension to the score on the other part on another dimension, the score 
is only trivially lower: .25. 
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Tablft 14 



Msan Cpm lotions B«twem Parts Whra Scores 

Ara Assigned By Different Reeders, 
vervont Portfolios And ZTBS standardisation 







Vermont 


ITBS, Disattenuated 




Raw 


Disattenuated 


Same 
Genre 


Different 
Genre 


Grade 4 
Same dimension 
Different dimensions 


.28 
.25 


.77 
.73 


.44 


.38 


Grade 6 










Same dimension 


.37 


.89 


»52 


.28 


Different dimensions 


.34 


.81 







When these correlations are "disattenuated/' however-that is, adjusted to remove 
the effect of the low agreement rate among readers-they become quite high. The 
average disattenuated correlation between parts on a single dimension was .77 in the 
fourth grade and .89 in the eighth grade.^ In other words, once one removes the effect of 
the generally low reliability, scores on one part are lairly strong predictors of scores on 
the second part, particularly in the eighth grade. 

These disattenuated correlations are surprisingly high in the light of experience 
with other direct assessments of writing, which typically show that ratings of student 
essays vary substantiaUy within genres and markedly across genres (e.g., Dunbar, 
Koretz, and Hoover, 1991). For example, the Iowa Tests of Basic Skills include an 
optional direct test of writing scored with a "focused holistic" rubric. When pieces were 
graded by different raters, the correlation of scores across different pieces within one 
mode of discourse averaged only .44 in fourth grade and .52 in eighth grade (after 
disattenuating for rater unreliability) in a standardization sample Across genres, the 
disattenuated correlations were .38 and .28, respectively (Hieronymus, Hoover, Cantor, 
and Oberly, 1987). For comparisoa these correlations are entered in the "different parts, 
same dimension" rows of Table 24. The Vermont portfolio program maintains only 
loose control over the content of portfolios, but the guidelines clearly call for the 

^These correlations were disattenuated by dividing the raw correlations by the square root 
of the product of the relevant correlations within piece and dimension but between raters. 

^ In the case of the ITB5, disattenuation was modest, because in terra ter reliabilities were 
very high, ranging from .88 t:> .99 depending on the grade level and the mode of discourse 
required by the prompt. 
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inclusion of different genres, so the lower of these ITBS correlations (between genres) are 
probably the more reasonable Ccmparison7 

A possible clue to the high disattenuated correlations between parts in Vermont 
can be found in the correlations between the part scores provided by a single reader. 
Although the raw correlations between parts scored by different raters were very low 
(before correcting for reliability), the correlatior\s within a single rater tended to be 
considerably higher. For example, for a single dimensioa correlations between parts 
scored by different readers were .28 and ,37 in the fourth and eighth grades, 
respectively, but they were about .60 when scores were provided by a single rater. (See 
Table 15; the between-rater correlations are repeated from Table 14 for comparison.) 
That is, a given reader's score on one dimer\sion for one part of a given portfolio predicts 
to a moderate degree the score that reader gave the other part on the same dimerxsiort 



Tmhlm 15 

Uman Correlation* B«tw««n Parts Wbw Scone 
Are Aseigned By 'thm Sane Aud Different Reader 





Same 


Different 




Reader 


Reader 


Grade 4 






Same dimension 


,59 


,28 


Different dimensions 


.44 


,25 


Grade 8 






same dimension 


.62 


• 37 


Different dimensions 


.49 


,34 



There are several possible explanations for the relatively high degree of 
corxsistertcy across parts within a single rater compared to the much lower consistency 
across raters. Ore possibility is that some readers have reasonably consistent rules for 
rating the portfolios but that different readers have different rules. For example, one 
reader nrught be more influerured by inappropriate use of commas than another,® This 

^The ITBS scores reflect a single focused holistic score for each piece, which would tend to 
make them more reliable than the dimension-«pecific Vermont scores. This makes the higher 
disattenuated correlations from the Vermont program even more striking. 

*In June of 1993, we observed many hours of discussions by raters working on "calibration 
pieces" that were used to increase the similarity of their ratings. There were a number of 
discussions about scoring discrepancies that fit this hypothesis-for example, raters disagreeing 
about how heavily to weight repeated "usage, grammar, and mechanics" errors of the same type 
within a single piece. We have no comparable observations of the 1992 scoring that yielded the 
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implies either insufficiently precise rubrics or insufficient training in their use. Another 
possibility is a halo effect-that is, a consistency of scores across parts that is imposed by 
the rater and that goes beyond the "true" consistency of performance on the parts of a 
students' portfolio. That is, one or more pieces in a portfolio may color some readers' 
evaluations of other pieces in the portfolio, so that scores on the later pieces resemble 
scores on the earlier pieces more than they should.^ These possibilities are not mutually 
exclusive. 

RELATIONSHIPS BETWEEN SCORES ON DIFFERENT DIMENSIONS 

A related issue is the degree to which the different scoring dimensions (five in 
writing, seven in mathematics) function independently. The use of multiple dimensions, 
like the inclusion of more than one part, has several rationales. Regardless of the 
patterns shown by scores, employing mxiltiple dimensions may be an effective way to 
focus instruction on desired attributes of student work. From the perspective of 
measurement, however, the value of multiple dimer\sions depends on the degree to 
which scores on different dimensions provide additional, indeuendent information 
about the quality of student work. If dimensions do not provide independent 
information, little would be lost-in terms of measurement-bye mploying fewer 
dimensions or by combining scores across dimensions before reporting them. 

In general, the scores assigned by a single rater showed considerable consistency 
across the scoring dimensions. For example, in fourth grade, raters assigned 58 percent 
of best pieces exactly the same scores on the Details and Organization dimensions (Table 
16). When scores were assigned by different readers, the rate of agreement was much 
lower: only 40 percent (Table 17), When a single rater assigned the scores, only 2 
percent of the students had a 2 or 3 point difference between their scores on details and 
orgaru2ation; when different raters assigned the scores, 10 percent of the students' scores 
differed by that much. 



data presented here, but it is reasonable to assume that such discrepancies were at least as large at 
that ime, 

^or example, one recent study of writing portfolios from a single elementary school found 
that scores assigned by teachers to entire portfolios tended to be overly influenced by the score 
assigned the highest-scoring piece in each (Herman, Gearhart, and Baker, 1993). 
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TabU 16 

D#gr«« Of AgrttttiMnt B«tw«M Th« ClASsrooa T««Gb«r'0 
Scores On thm Detail Kad organlSAtloa DlMnslons 

(Tabled values are the percentage of students with each 
of the possible combinations of scores on Organization 
and Details) 



Organization 







1 


2 


3 


4 




1 


1 


2 


0 


0 


Details 


2 


2 


16 


14 


1 




3 


0 


8 


27 


8 




4 


0 


1 


6 


14 



Note: Percent of cases with exact agreement: 
(1 + 11 + 21 + 7 = 40%) 



Table 17 

Degree of Agreement Betireen The Classroon Teacher's 
Mtails Score A.^ The Independent Reeder's 
Organ! set ion Score 

(Tabled values are the percentage of students with each 
of the possible combinations of scores on Organization 
and Details) 







Organization 


( 2nd Reader) 


1 


2 


3 4 


Details 


1 


1 


1 


1 0 


(Teacher) 


2 


2 


11 


15 5 




3 


1 


11 


21 10 




4 


0 


3 


10 7 



Notet Percent of cases with exact agreement: 
II + 11 + 21 + 7 = 40%) 



The degree to which readers assign the same scores across dimensions is even 
clearer when correlations are used to describe agreement. The correlations of scores 
across dimensions are very similar to the correlations across parts discussed earlier. For 
exa Tiple, in fourth grade, the average correlation between the scores on two dimensions 
assigned by one rater to one part was .57 (Table 18). By way of comparison, the 
correlation between the scores assigned to the two parts on one dimension by one rater 
averaged .59. {The correlations between parts in Table 18 are repeated from Table 15 for 
comparison.) The corresponding agreement rates when scores were compared across 
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two rea > were much lower (less than .30 in fourth grade), but again the correlations 
across dimensions were nearly identical to the correlations across parts. (Eighth grade 
results, omitted from Table 18 for simplicity, were very similar, excep* that the 
correlations between different raters were about 0.10 higher than the corresporxling 
correlations in the fourth grade). 

Tabla IS 

Maan Corralationa Among DiMnfliona Whan flcoraa 
Are Aaaignad By Tha Saaa And Diffarant Eaadar,. 
Orada 4 





Same 


Different 




Reader 


Reader 


Same Part 






Same dimension 


(1.00) 


.34 


Different dimensions 


.57 


.29 


Different Parts 






Same dimension 


.59 


.28 


Different dimensions 


.44 


.25 



One would expect agreement to be lower when scores are provided by two 
different raters because of the unreliability of ratings documented above; the key 
question is how much lower. To gauge the relative size of these correlations, one can ask: 
what set of scores provides the best prediction of another? For example, say that one 
wanted to predict the scores assigr«d by one rater to one part on a specific dimension. 
Ideally, the best predictor would be the scores assigned to the same part on the same 
dimension by another rater; that would indicate that the quality of performance on that 
part and dimension, rather than idiosyncracies of raters, determined scores. That was 
far from the case in Vermont. In the fourth grade, the correlation of scores for the same 
part and dimension but across readers was only .34, meaning that on average, scores 
from one reader predict only about 12 percent of the variance in scores given to that part 
on that dimension by the other rater (.342 = 12). In contrast the scores assigned to that 
part by the same rater but on a different dimension predict about 32 percent of the 
variance in scores (.57^ s ,32). Indeed, it would even be better to know how the same 
rater scored the other part on a different dimension (.44^ = 19% of the variance) than to 
know how a different rater scored the same part on the same dimension. 

When adjustments are made for the low rate of interrater agreement, the 
correlations between dimensions become extremely high. The most reliable correlations 
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between dimensions that we can obtain are those based on the sum of the two raters 
scores for each twice-scored portfolio. On the average, the correlation between two 
dimensions was about .69 when the score on a dimension was the sum of the two 
readers* scores on that dimension across the two parts (Table 19). Taken at face value, 
these correlations would suggest a moderate degree of independence between 
dimensions, but much of that apparent independence is simply an artifact of random 
rating error. Virtually all the correlations soar to the middle or upper .90s (close to 
perfect) when they are corrected for the less than perfect agreement among readers 
(Table 20). It is therefore likely that any difference between a student's scores on two 
dimensions results from random error and is not meaningful or interpretable. These 
findings suggest that in 1991-92, it would have been nearly as useful to assign only a 
single score to each part of the portfolio because the scoring did not reliably distinguish 
among the different dimensions. 



Tabltt 19 

Zntttr-DiMMion Comltttiona 
writing, U&«dju«t«d 





Purpose 


Organization 


Detai Is 


Voice/Tone 


U/G/M 


Purpose 




.84 


.78 


.74 


.60 


Organization 


.84 




.75 


.69 


.67 


Details 


.81 


.79 




.76 


.57 


Voice/Tone 


.74 


.73 


.78 




.53 


U/G/M 


.73 


.77 


.70 


.64 




NOTE: Data 


for Grades 


4 and 8 appear 


belov and 


above the 


main 



diagonal, respectively. 



Tabl* 20 

Zntar-DiMMion Corr«l«tiontt 
writing, Adju«t«d 





Purpose 


Organization 


Detai Is 


Voice/Tone 


U/G/M 


Purpose 






♦ 


♦ 


♦ 


Organization 


♦ 




* 


* 


* 


Details 


♦ 


* 




• 


♦ 


Voice/ Tone 


• 


♦ 


• 




.94 


U/G/M 


• 


♦ 




.92 




NOTE: Data 


for Grades 


4 and 8 appear 


below and 


above the 


main 



diagonal , respectively . 

* Adjusted estimate of correlation is greater than 1.00 
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SOURCES OF VARIATION IN SCORES: A GENERALIZABILITY ANALYSIS 

We used a slalisKcal technique called "generalizability * analysis (Cronbach, 
Gleser, Nanda, and Rajaratnam, 1972) to investigate the sources of variation in the scores 
given to students' portfolios. We estimated the degree to which the variation in scores 
could be attributed to three factors (students, readers, and part) and interactions among 
them. (See Appendix B for a brief discussion of generalizability analysis and more detail 
on the aruilyses summarized here.) 

Consistent with the results reported above, the genera lizability analysis foimd that 
much of the total variation in scores can be attributed to rater error. More specifically, 
over one half of the total variance in scores was attributable to non-systematic 
differences between raters. This appears as the combination of Reader x Student, Reader 
X Part, and residual variance components in Table 21. 

Table 21 

Parcant*g« Of Variance In Writing Portfolio Scores 
On K Typical Dimtneion That Wae Attributable 
To Varioue Pactore 



Source 


Grade 4 


Grade 8 


Students 


28 


36 


Students x Parts 


7 


7 


Readers 


13 


8 


Readers x Students 


18 


18 


Readers x parts 


1 


1 


Residual 


33 


31 


TOTAL 


100% 


100% 



Only a small part of the remaining variance in scores (about 13 percent in grade 4 
and about 8 percent in grade 8) can be attributed to the tendency for some raters to be 
systematically more lenient than others (the Reader effect in Table 21). As noted earlier, 
these small differerKes in leniency were largely unrelated to whether the reader was the 
student's own classroom teacher. 

Consistent differences among students or between parts accounted for less than 
half of the total variance in scores. Only about a third of the total variance in scores (28 
percent in the fourth grade and 36 percent in the eighth grade) can be attributed to 
differences amor^ students (Table 21). An additional 7 percent of the total variance in 
scores was due to the readers agreeing that the student did better on one part than on 
the other part (the Student x Part interaction). This is consistent with the finding 
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reported earlier that there was hardly any difference in mean scores between parts 
(Table 11). 

This analysis thus provides another index of the severity of tinreliability: the 
majority of the variance of scores arose because of disagreements among readers in their 
evaluations of the quality of portfolios. Moreover, the analysis further suggests that as 
of 1991-92, the two parts of the writing portfolio were for the most part not functioning 
independently, and little additional information was gained by including both parts 
rather than one. 

STRATEGIES FOR IMPROVING SCORE RELIABILITY 

The analyses described above found that the score assigned to a part on a given 
dimension cannot be trusted to provide an accurate measure of that part's qtjality on 
that dimension. This problem stems from the very low agreement rates between readers 
on what score to assign. This section discusses a number of ways to improve reliability: 
combining scores across dimerxsions, parts, and readers and adding additional parts or 
readers. (There are, of course, other strategies that might be used to increase reliability, 
such as better training and calibration of readers.) 

Effect of Averaging Scores Across arts and Dimensions 

More reliable scores for a student can be obtained by combining scores across 
parts or dimensior\s. Both types of combining in theory contribute additional, 
independent information about performance, which should make scores more reliable. 

However, in the 1991-92 writing data, even combining across both readers and 
dimensiorxs increased reliability orUy modestly. The resulting scores were still too 
unreliable to warrant confidence that they provide an accurate index of the quality of a 
given student's work. The reason that the improvement was modest was the sizable 
correlations between parts and dimerxsions (within a reader) discussed earlier. That is, 
because the parts and dimensions were substantially correlated, they provided orUy 
limited independent information, so combinirxg them did not greatly improve reliability. 
For example, Table 22 shows that the average correlation between two readers on one 
dimension on one part at Grade 4 was orUy .34. The correlation between readers 
increases to .39 on a single dimension if the scores on the two parts are averaged, i.e., if a 
student's score on a dimension is the mean of his or her best and rest scores on this 
dimereion. Combining over all dimensions and parts increases the correlation to .49. 
The same pattern was obtained at Grade 8, but the correlations were all slightly higher. 



There is little statistical disadvantage to combining scores across din\ensions and parts 
because as noted earlier, the separate dimension scores are not really interpretable and 
only a very small portion of the differences in scores between students stems from some 
of them doing better on one part than on the other part* However, combirung in this 
fashion would eliminate the as yet tmrealized potential for obtaining information about 
different aspects of a student's work. 



Table 22 

Averatge Correlation Between Reader* With 
Different Types Of coabiniag, Writing 



Type of Score 


4th 


8th 


No combining- -one dimension on one part 


»34 


.43 


One dimension--mean over both parts 


.39 


.49 


One part--mean over all 5 dimensions 


• 45 


-56 


Total — mean over all dimensions and parts 


.49 


.60 



Increasing the Reliability of Total Scores by Adding Parts or Readers 

If a total score is to be computed for each portfolio, its reliability will be influenced 
by the number of parts included in the portfolio: adding more parts will rruke the total 
score more reliable. Similarly, combining scores across additional readers will boost 
reliability. This section discusses the effects of adding parts and readers. 

In this context, reliability is the correlation that would be found between different 
portfolios produced by the same student. That is, each portfolio is considered to be a 
limited sample of the students work, and each portfolio is treated as only one of many 
portfolios that could have been constructed for that student. We have only a single 
portfolio for each student, but we estimated the correlation that would have been found 
among different portfolios from the same student by applying standard statistical 
methods to the scores assigi«i by the readers (see Appendix B). The resulting estimate 
is expressed as a correlation (reliability) cc efficient. The higher the coefficient, up to a 
maximum of 1.00, the stror^ger the estimated relatiorxship between the scores that would 
be assigned to different, but representative samples of the student s work. 

Increasing the number of readers who evaluate a portfolio from 1 to 2 has a small 
but rwticeable effect on the reliability of total portfolio scores (as defined above) 
regardless of the number of parts (pieces in ihe portfolio) that are evaluated separately. 
However, there is not much to be gained by having more than 2 readers. Figure 1 shows 
the estimated relationships between number of readers per portfolio, number of parts 



70 



50 



evaluated, and reliability for one of the writing dimensions (Similar figiires for all of the 
writing dimensions are included in AppeiKlix B.). As one benchmark for interpreting 
these data, a measure should have a reliability of .90 or higher before scores on it are 
used to make important decisiorxs about individual students (as distinct from larger 
units, such as schools). Most standardized achievement tests have reliabilities that 
satisfy this criterion 



Grade4.Detato 




0 I I 1 I I 

1 2 3 4 S 

No. of Ralen 



Figure l*-Bf£«Gta on r«liability of incrMclng thm UMmhmr of rater* or 

pi«G««; grade 4, data! la 

Increasing the number of parts within a portfolio that are evaluated improves 
reliability, but with rapidly diminishing returr\s. There is r\ot much to be gained by 
having each reader evaluate more than 5 parts, largely because of the strong correlations 
among parts discussed previously. This trend holds for any given number of readers. 
Put another way, it would be difficult to justify having more than two indepetvdent 
readers per portfolio aiwl having them assign scores to more than five separate pieces in 
that portfolio. However, even under these conditions, reliability is still quite low (.65 to 
.70) for the purposes of making decisions about individual students, particularly given 
the amount of reader time that would be required to achieve this level (a teacher 
evaluates only 2 to 3 portfolios per hour). 
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COMPARISON TO THE UNIFORM TEST 

In addition to preparing their portfolios, 4th and 8th grade students also took an 
on-demand writing test. The degree of agreement among readers in grading the 
responses to this test was much higher than it was amor^ those who graded the 
students portfolios. This section discusses these differences in reliability and speculates 
on some of the reasons for them. 

The "uniform" exam had a single essay question at each grade level and all the 
students took this test under the same standardized corvlitions. Student responses were 
scored by readers other than Vermont teachers who were hired by a private contractor. 
These readers used the same five dimensions as were used to grade the portfolios. A 
sample of 695 fourth graders and 608 eighth graders had their answers read twice. Arvi 
as with the portfolios, the second reader did not know the score assigned by the first 
reader. 

Inter-reader agreement levels on the uniform test were much higher than they 
were on the portfolios. The mean correlations at Grades 4 and 8 fDr the total score across 
all five dimensions were .87 arvi .82, respectively. In contrast, the mean correlation 
between two readers in total portfolio scores (i.e., across all dimerwioi\s arul parts) was 
only .49 at Grade 4 and .60 at Grade 8 (see Table 22). 

Why were the uniform readers so much more consistent with each other than 
were the portfoUo graders Each portfolio had 6 to 8 samples of the student s work 
compared to orUy a sir^le essay ar\swer on the uniform test. Thus, agreement should 
have been higher on the portfolios because there was more opportunity for charvre 
factors to be balanced out. The students who took both measures came from essentially 
the same populations, so this was not a source of the disparity either. 

A more likely explaiution is that high agreement rates can orJy be obtained when 
all students respond to the same or similar prompts or when they all produce works that 
fall within certain well defined genres, e.g., each portfolio contains one poem, one short 
story, etc. That did not happen in Vermont in 1992. As a result, portfolio readers were 
asked to assess whether one student's response to one task was better than another 
student s response to a totally different task. This job would challet^e even the most 
cor\scientious and skilled grader, and given the results discussed above, we ru)W have 
oegun to question whether it can be done with an adequate level of coi\sisterKy in an 
operational program. 
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CONCLUSIONS 

The findings above led to the following concliisions regarding the reliability of 
writing portfolio scores in both grade 4 and grade 8: 

1) Most readers appear to form a general impression of a portfolio's quality, aiKl 
this impression seems to affect the scores they assign to all dimensions and 
parts. However, different readers often develop very different impressions of 
a portfolio's quality. Consequently, there is a moderate to high degree of 
consistency across parts aiKl dimensions in the scores a given reader assigr\s, 
but a very low degree of agreement between readers in their evaluation of a 
portfolio's quality. 

2) The foregoing sittjation has several ramificatiorts. For instance^ any 
difference in scores among dimensions within a student's portfolio is more 
likely to be due to chance than to any true difference in student proficiency 
levels. Thus, any differeiKes in scores between dimensions in a portfolio are 
not meaningful. The same is true for differences in the scores assigned to the 
different parts of the portfolios. 

3) On average, a student's own teacher was only very triviaUy (avd not 
statisticaUy significantly) more lenient than was the independent reader. 
Thus, this potential bias was not an important ir\fluence on scores. 

4) On the average, the score assigned to the best piece was only slightly higher 
than the score assigned to the rest of the portfolio. This pattern suggests that 
in terms of measurement, there was no functional difference between the best 
aiKi the rest, and there was nothing gairied (in tejms of measurement) from 
identifying one piece as the "best" one. 

5) Similarly, scores were substantially correlated across dimer\sions, and 
therefore including scores on the five dimensions added relatively little 
ir\formation about studeits' performance. 

6) Using two indepeiKlent readers per portfolio and separately grading as many 
as 5 different pieces in a portfolio will probably improve the reliability of 
total scores, but increasing the number of raters or pieces yet hirther would 
probably produce only minor improvements. Moreover, even basing scores 
on two ratings of five pieces would not produce a sufficiently reliable score 
for making decisions about individual students uiUess rater agreement is 
increased markedly. 
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Scores on the uniform test shewed much higher rater reliability than did 
portfolio scores. We suspect that a major reason for this difference is the fact 
that the tmiform test required all students to responded to the same question 
(so that raters could be trained specifically to score responses to that prompt). 
However, fxirther investigation is needed to separate the effects of 
standardization of tasks, training differernres, and other factors on the 
reliability of portfolio scores. 
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4. RELIABILITY OF MATHEMATICS PORTFOLIO SCORES 

This chapter reports on the reliability of the scores assigned to mathematics 
portfolios. As in the previous chapter, we focus primarily on the degree of agreement 
between readers, but we also examine the extent to which the quality of a student's work 
was consistent across dimensions and across the different pieces in the portfolio. 

We foimd that the readers who graded the mathematics portfolios often disagreed 
with each other about which score should be assigned. In addition^ there was usually 
less consistency among the pieces in a student's mathematics portfolio than between the 
parts of a writii\g portfolio. Consequently, the reliability of mathematics portfolio scores 
was no better than that of the writii\g portfolio scores.l 

CHAPTER OVERVIEW 

Tlus chapter addresses many of the same questions as the preceding one, and it 
largely follows the same organization. The next portion describes the procedures used 
to gather ir\formation about the degree of agreement between readers. We then discuss 
the rates of agreement between readers on individual pieces and dimensior\s. 
Subsequent sections examine the relationships amon^^ dimensions and pieces, estimate 
the contribution of various factors to the total variation in scores, and discuss the impact 
of alternative strategies for improving reliability. Issues discussed in the previous 
chapter that are not germane to this one include possible bias in teachers' ratings, 
systematic differences in scores among pieces, and comparisons to the reliability of 
uruform test scores.^ 

PROCEDURES 

Mathematics portfolios were scored somewhat differently than writing portfolios. 
There was no distiiKtion between "best" and "rest" in mathematics. Rather, each 
mathematics portfolio included 5 to 7 best pieces, and each of these pieces was graded 

^One measure-simple percentage agreement between raters-appears better for 
mathematics than for writing, but we explain below why that is probably misleading. 

^assroom teachers were encouraged to score their own students' portfolios, but those 
scores were not used in the state reporting system evaluated here. In mathematics, all scored 
pieces were considered best pieces, and pieces were identified only by arbitrary position in the 
portfolio, so there is no reason to expect differences in average scores among them. And the 
uniform test in mathematics, unlike that in writing, was multiple choice, so its "rater reliability^ 
was perfect. 
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on a 4- point scale on each of seven dimensions. Three dimensions were classified as 
pertaining to communication (language of mathematics, mathematical representations, 
and presentation) and were labeled CI through C3. Four dimensions were classified as 
aspects of problem solving and were labeled P51 through PS4 (see Appendix C for 
details). Thus, a reader assigned 35 to 49 scores to a portfolio (depending on the number 
of pieces) before grading the next portfolio. 

The analyses described below were conducted on 803 Grade 4 portfolios and 344 
Grade 8 portfolios that had at least 5 pieces scored twice. The second reader did not 
know the scores assigned by the first reader. Unlike writing, neither reader was the 
student's own classroom teacher. The 99 readers of Grade 4 portfolios and the 46 
readers of Grade 8 portfolios participated in a training session before they began 
assigning final scores. 

In analyzing the reliability of mathematics scores, we used two composite scores 
that were not used by the State for its reporting. First, we computed the average score 
for the portfolio on each dimension, across all pieces with valid scores. Second/ we 
computed total score for the portfolio by summing these average scores across the seven 
dimensions.^ These scores were computed separately by reader. 

AGREEMENT BETWEEN READERS 

Because neither of the readers of mathematics portfolios was the students' own 
teacher, there was no opporttmity to examine possible biases on teachers' evaluations of 
their own stxadents. Agreement between readers was therefore only a question of the 
degree to which they were consistent with each other in deciding which pieces 
warranted high or low scores. 

The degree of agreement between readers was more variable in mathematics than 
in writing, but overall, it was quite low on both types of portfolios. This section presents 
the data on agreement rates on individual pieces, dimensions, and total mathematics 
portfolio scores. 

Percent Agreement On Pieces 

The degree of agreement between two readers on a single piece was generally 
higher in mathematics than on a single part in writing, but this difference is misleading 

^The State used a complex algorithm, rather than a simple averaging, to obtain a composite 
score on each dimension. In this chapter, we focus on averages because they are simpler and 
more reliable. In Chapter 5, however, we use the state's composite scores, because the focus there 
is quality of aggregate scores that would be based on those composites. 
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because the iratheniatics agreement rates were still not much better than what would be 
expected by chance alone. That is, the rates of agreement were not much better than one 
would expect if each rater's score on one piece had been compared to a randonUy 
selected piece that would usually be from a different student. As voted in Chapter 3, the 
rate of agreement expected by chance iiKseases as scores become increasingly 
concentrated at one or two points on the scale, and mathematics scores tended to be 
more highly coiKentrated than writing scores. 

Table 23 illustrates a typical pattern by showing the scores assigned by different 
readers to pieces on dimension PS2 C'How: Procedures") at Grade 4. The rows 
correspond to the scores assigned by the first reader and the columns to the scores 
assigned by the second reader. The entries in the table are the percentage of all twice- 
scored portfolios receiving each of the 16 possible combinatiorxs of scores. For example, 
4.5 percent of the pieces that were read twice received a score of 1 from both readers. 
Overall, the two readers agreed on the score that should be assigned to 55.5 percent of 
the pieces (45 + 7,1 + 43.5 + 0.4 = 55,5), The 55 J percent figtare indicates readers agreed 
with each other slightly more than half the time on the score that should be assigned. 

However, becaiise scores were highly concentrated, the observed 55.5 percent rate 
is not much better than one would expect to occur by chance. Specifically, the likelihood 
that both readers would assign the same score to a piece by chance is the product of the 
rates at which they each assigned that score. In Table 23, for example, the likelihood that 
a piece woukl receive a score of 1 from both readers by chance is .124 x .119 = .015 or 1.5 
percent. Similarly, the likelihood that both would assign a score of 2 is ,233 x .219 « .049 
or 4.9 percent. The correspondii\g cafculation is made for the other two score levels and 
the sum of the four products (1.5 + 4,9 + 38.4 + 0.1 » 44.9) is the overall rate of agreement 
that is likely to arise by chance. In Table 23, the chance rate (44.9 percent) is only 10.6 
percentage points less than the observed rate of 55.5 percent. 
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T«bl« 23 

Perc«nt«g« of Orad« 4 lUthoiAtictt Pimemm Rac«iving |«ch 
Combination of Scoraa on tha PS2 Dinenaion 



Score Assigned by Second Reader 





1 


2 


3 


4 


Total 


Score 


1 


4.5 


3.5 


4.3 


0.1 


12.4 


Assigned 


2 


3.4 


7.1 


11.1 


0.3 


21.9 


by First 


3 


3.9 


12.5 


43.5 


2.6 


62 .5 


Reader 


4 


0.1 


0.2 


2.5 


0.4 


3.2 


Total 


11.9 


23.3 


61.4 


3.4 


100.0% 



The rate of agreement between two readers on a piece varied across dimensions 
(Table 24). They ranged from 41 percent CWhy: I>edsions" at Grade 4) to 86 percent 
("What: Outcomes'* at Grade 8). However, the higher rates of agreement typically 
reflected more substantial concentration of scores rather than evidence that raters could 
differentiate reliably among pieces of different quality; that is, even the higher rates of 
agreement were not much better than expected by chance. 



Tabl« 24 

Mean Actual and lxp«ct«4 by Ch«nc« Kgxmmnmnt Rat«« on 
Uatheaatictt Pi«c« scorM, by or«d« and Diaanttlon 



Dimension 


Grade 


4 


Grade 


8 


Actual 


Chance 


Actual 


Chance 


Cl -Language of Math 


54 


42 


50 


39 


CI -Math Representations 


48 


33 


52 


33 


C3-Presentation 


45 


33 


45 


30 


PSl-Understanding of Task 


66 


57 


65 


55 


PS2-How: Procedures 


55 


45 


59 


45 


PS3-Why: Decisions 


41 


31 


42 


31 


PS4-What: Outcomes 


80 


75 


86 


81 


Average 


56 


45 


57 


45 


NOTE: Agreement rate = percentage of 


pieces 


that received 


the same 



score from both readers on a dimension. 



The most extreme concentration of scores was on dimension PS4 ("What: 
Outcomes of activities"). Table 25 shows that 86 percent of the Grade 8 pieces received 
the same score from both readers on this dimension. However, this seemingly high rate 
of agreement occurred because nearly all the eighth grade pieces were given a score of 1 
on this dimer\sion. Specifically, 89 percent of the students were given a score of 1 by the 
first reader, 90 percent were given a score of 1 by the second reader, and 83 percent 
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received a score of 1 from both readers. A few other criteria also had high degrees of 
score concentration (but not as high as the one in Table 25). For example, about 71 
percent of the Grade 8 students were given a rating of 3 on the "understanding of task" 
dimensioa 



Table 25 

Psrcsntags of Orads 8 Xathaiiiatics Piscsa Receiving Saeh 
Coabination of deoraa on tha fiS4 Diaanaion 





Score 


Assiqned by 


Second 


Reader 


Total 


1 


2 


3 


4 


Score 


1 


83.2 


4.9 


0.7 


0.1 


88.9 


Assigned 


2 


6.2 


2*6 


0.7 


0.0 


9.5 


by First 


3 


0.8 


0.3 


C .3 


0.0 


1.4 


Reader 


4 


0.1 


0.1 


0.0 


0.0 


0.2 


Total 


90.3 


7.9 


1.7 


0.1 


100.0% 



A high rate of agreement when scores are highly concentrated may be ue to 
reliable scoring or chazKe. In the case of PS4 'IVhat: Outcomes/' chance would have 
produced an overall agreement rate of 81 percent, oiUy trivially lower than the 86 
percent actual agreement shown in Table 25.^ This is aruilogous to throwing darts at a 
target: if the bulls^ye is made large enough that almost all daris hit it, a high proportion 
of bulls-eyes no longer indicates which players can throw darts accurately. In the case of 
the PS4 dimension, a score of 1 is the bulls-eye, and the fact that almost all pieces get a 1 
does not indicate whether readers can reliably differentiate among pieces deserving 
scores of 1, 2, 3, or 4. 

Jo sum up, two readers often disagreed with each other on the score that should 
be assigned to a piece on a given dimer\sioa When they did agree with each other 
frequently, it was because of extreme concentration of scores at otw or two score levels. 
On all seven dimensions, the degree of agreement between readers was only slightly 
greater than what might have occurred by chance alone. In short, the data provide no 
evideiKe that raters could distinguish reliably between pieces that differed in quality on 
any of the seven dimerxsions. 

Similarly, the conditional probability that a student will receive a score of 1, given that 
another rater has already assigned a score of 1, is .9^nly trivially different than the 
unconditional (overall) probability of .90 percent. 
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Percent Agreement On Dimensions 

Up to this point, we have discussed the degree of agreement between two readers 
on a single piece in a mathematics portfolio. We now examine agreement rates between 
readers on a single dimension for the portfolio as a whole. Were the readers consistent 
in saying that on a particular dimension, a given portfolio was one of the best, one of the 
worst, or somewhere in the middle relative to all of the other portfolios they graded? 

To investigate this issue, we computed each portfolio's mean score (over its 5 to 7 
pieces) on a dimension. Next, we rank ordered the means assigned by the first reader on 
this dimension from the highest to lowest, and then divided this distribution into four 
equal parts-the highest 25 percent (i.e., the top quartile), the next highest quartile, etc. 
We then repeated this process for the second reader. 

These calculations, which were done separately for each dimension, allowed us to 
examine the degree of agreement between readers on a dimension. If readers agreed 
perfectly with each other regarding the grade that should be assigned to a student on a 
dimension, then every student's quartile on the first readii^ on that dimension would be 
the same as that student's quartile on the second readii^-l(X) percent agreement. 
However, if there was no oor\sistency between readers on this dimension, then by 
chance, 25 percent of the students would still be in the same quartile on both readings. 

There was very little agreement between readers on a dimension even though 
each dimer\sion score was based on 5 to 7 pieces. Table 26 illustrates the typical pattern. 
The first column shows that at Grade 4 on the "C2 - Mathematical Representation" 
dimension, only 36 percent of the students were in the same quartile on both readings 
(i.e., only 11 percentage points better than chance)-64 percent changed odb or rr»re 
quartiles. In fact, 5 percent of the portfolios changed three quartiles, i.e., they went from 
the very bottom quartile on one reading to very top quartile on the other reading. 

Tabla 26 

Poreantogo o£ aradf* 4 atudanta imoaa Ralativa Standing Changed 
0, X, 2, or 3 Quartilaa Hhan m Diffarant Raadar Oradad tha Portfolio; 
Raaulta for a Typical Diaanaion and Total Score 



Ajnount of Change 


Typical Dimension^ 


Total 
All 


Score Across 
Dimensions 


No change 


36% 




45% 


1 Quartile 


37 




43 


2 Quartiles 


22 




10 


3 Quartiles 


5 




2 



Mathematical representation. 
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Percent Agreement On Total Scores 

The last column in Table 26 shows that even when the analysis is based on total 
scores across all pieces and dimensions, 55 percent of the students charged one or more 
quartiles between readings. Taken together. Tables 23 * 26 show that the degree of 
agreement between readers on pieces, dimensior^, and total scores is only slightly better 
than what is likely to occur by chance alone. 

Correlation Coefficients 

The correlation coefficient is another measure of the extent to which readers 
agreed with each other in their assessment of the relative quality of students work . 
According to this index, readers often disagreed with each other as to which students 
did better than others on a piece, on a dimension, or even the whole portfolio. 

Unlike simple agreement rates, correlation coefficients are not inflated by the 
readers assigning the same score to almost all the pieces. For example, because almost 
all the pieces received a score of 1 on the PS4 ''What: Outcomes" dimension in grade 8, 
the readers agreed with each other 86 percent of the time on the score that should be 
assigned. However, the correlation coefficient between two readers on a piece on this 
dimension was orUy .30. This is actually slightly below the typical correlation between 
readers at the piece level on the other dimensions (see first two columns of Table 27). 

Tabl« 27 



Corralationa B«twMn 


Raadara 


on Pittctts 


and DiMnaiona 




Piece 


Level 


Dimension Level 


Dimension 


Grade 4 


Grade 8 


Grade 4 Grade 8 


CI -Language of Math 


.34 


.32 


.30 .28 


Cl-Math Representations 


.41 


.47 


.36 .34 


C3 -Presentation 


.39 


.45 


.51 .53 


PSl -Understanding of Task 


.30 


.32 


.42 .38 


PS2-How: Procedures 


.33 


.36 


.48 .38 


PS3-Why: Decisions 


.35 


.37 


.48 .37 


PS4-What: Outcomes 


.30 


.31 


.43 .39 


Mean 


.34 


.37 


.42 .38 


NOTE: The Piece Level columns show 


the correlations between the 



first and second readers in the scores they assigned to individual 
pieces. The Dimension Level columns show Khe correlations between 
readers in the mean scores they assigned to a portfolio on a dimension 
where a dimension score is the mean over 5 to 7 pieces. 
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Combining information across pieces increased the correlations between readers, 
but only marginally. As noted earlier, each portfolio's mean score on each dimension 
(across its 5 to 7 pieces) was computed separately by reader. The correlations between 
the means assigned by the first and second readers on a dimension ranged from .28 to 
.51 (see last two columns of Table 27), Averaged across all dimensions, the means of 
these dimension- level correlations between readers were .42 in the fourth grade and .38 
in the eighth grade, only slightly higher than the corresponding correlations at the level 
of individual mathematics pieces (.34 and .37). These dimension-level correlations were 
quite similar to those found for writing: .39 in the fomth grade and .49 in the eighth 
grade. 

Summing scores across all dimensions to yield a single total score per portfolio 
improved reliabiity modestly relative to the dimension-level scores, but even thse total- 
score correlations were only moderate. At grade 4, the total score assigned to a portfolio 
(across all dimensions) by one reader correlated .60 with the total score assigned by the 
other reader. At grade 8, the correlation was .53. These values are similar to the 
corresponding correlation coefficients on the writing portfolios (.49 and .60, 
respectively). 

RELATIONSHIPS AMONG SCORES ON DIFFERENT PIECES 

All of the scored pieces in the mathematics portfolios were considered best pieces. 
Moreover, there was nothing to distinguish one piece from another beyond their 
arbitrary positions in the portfolio. Thus, this section considers only the correlations in 
scores across pieces. 

Scores for the various pieces of a mathematics portfolio typicaUy had lower 
correlations with each other than did the scores on the two parts of the writing 
portfolios. The simple correlations between pieces were extremely low when they were 
scored by different raters. Across dimensions, the correlations were .10 or less, and 
within a single dimension, they were .13 or lower. (These are the raw correlations in 
Table 28.) Even when scores were assigned by the same reader on the same dimension, 
the correlations between pieces of the math portfolio were low, averaging only .27. In 
contrast, the corresponding raw correlations in writing were all two to three times as 
large (Table 14). 
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Hmmn ohmmwd and Diaattanuatsd Corralationa fi«tw«an Mmthftmtiea 
Pi«eaa vh«n Scoraa Ara Aaaigned by tha Sana or Diffarant Raadara 

Grade 4 Grade 8 

Raw Corrected Raw Corrected 

Different Readers 

Same Dimension .13 .39 .12 .33 

Different Dimensions .10 .?9 .09 .27 

Same Reader 

Same Dimension .27 — .26 

Different Dimensions :_19 .21 

NOTE: Disattenuated correlations cannot be computed for the same 
reader. 

The mtich lower correlations in mathemaKcs compared to writing cannot be 
attributed to differences in reader reliability. When disatteniaated for reader reliability, 
the mean correlation between readers on a mathematics piece in Grade 4 was .39 for the 
same dimension and .29 across different dimensions. Similar correlations were obtaii>ed 
at Grade 8. The corresponding correlations in writing were .77 and .73 in the fourth 
grade and .89 and .81 in the eighth grade (Table 14). 

It is not clear why the correlations in mathematics were much lower than they 
were in writing, but several related possibilities present then\selves. Mathematics pieces 
within a portfolio may have been, on average, more dissimilar to each other than were 
the pieces comprising writing portfolios. Some mathematics problems, incontrast to 
essays, may have relatively clear correct answers (in the Vermont program, "clear 
solutions" or "dear ways of presentation" might be more appropriate). Although 
phrased in generic langxiage, the scoring rubrics may have been more clearly applicable 
to some pieces within a portfolio than to other pieces. The available data were not 
sufficient, however, to evaluate these or other possible explanations. 

RELATIONSHIPS BETWEEN SCORES ON DIFFERENT DIMENSIONS 

Mathematics scores for a given piece showed low correlations across dimensions. 
Regardless of grade level, the average correlation between dimensions for a single piece 
was roughly .20 when different raters assigned the scores, and .35 when the same rater 
assigned the scores (Table 29). 
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Tablft 29 

Umma Corr«latioM B^twsm DlMMions Hhmn fleortt* Ar« 
Assigned by thm Smam V^rsua Diffarsnt Madsrs 





Mathematics 


Writing 




Grade 4 Grade 8 


Grade 4 Grade 8 


Different Reader 


.20 .20 


.29 .37 


Same Reader 


.35 .36 


.57 .61 



Even though correlations between dimensions at the individual piece level were 
typically low, the data suggest tha t many of the dimensions may not be independent at 
the level of the total portfolio. The most reliable score we could obtain on a dimension 
for a mathematics portfolio was produced by taking the mean of all of the scores 
assigned to it-that is, the scores assigned to all pieces by both raters. These correlations 
averaged a bit over .50 (Table 30). When these correlations are disattenuated for rater 
unreliability, all but one of the dimertsions (PS4 - "What outcomes of activities") were 
highly correlated with each other, with many correlations above .90 (Table 31). These 
correlations suggest that with the exception of PS4, there was not much underlying 
difference among the dimensions in the 1991-92 total portfolio scores. However, this 
extreme a correction for disattenuation is risky, and these disattenuated correlations are 
only uncertain estimates of what vw>uld have been found if raters scored reliably. 

TabU 30 

Int^r-diMDsion Comlations 
Math, Unadjusted 



CI C2 C3 PSl Pg2 PSj PS4 

CI .41 .63 .58 .58 .55 .43 

C2 .36 .47 .47 .53 .46 .32 

C3 . 57 .55 . 72 .75 .83 .39 

PSl .45 .51 .73 .81 .72 .39 

PS2 .50 .56 .75 .79 .78 .35 

PS3 .57 .50 .81 .66 .77 .29 

PS4 .36 .34 .36 .36 .35 .33 



Note: Data for grades 4 and 8 appear below and above the main 
diagonal , respect ively . 
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Tabla 31 

Intttr-diMMion Corralation* 
Math, Adjuitcd 



CI C2 C3 PSl PS2 PS3 PS4 

CI .87 * * * * .86 

C2 .73 * • 1.00 .89 .59 

C3 * .93 • * * .62 

PSl .86 .92 • * * .71 

PS2 .92 .95 • * * .63 

PS3 • .88 • * • .53 

PS4 .68 .59 .60 .56 .54 .33 

* = Adjusted estimate of correlation is greater than 1.00. 



Note: Data for grades 4 and 8 appear below and above the main 
diagonal, respectively. 

SOURCES OF VARIATION: A GENERALIZABILITY ANALYSIS 

As with writing portfolios, we used generalizability analysis to explore how much 
of the total variation in mathematics portfolio scores was attributable to various sources: 
students, readers, pieces, interactions among these factors, and "noise" (residual error 
variance). With one very important exception, the results in mathematics paralleled 
those in writing. Specifically, just as in writing: 

• About half of the variance in mathematics scores on a dimension was due to 
unsystematic inconsistencies between readers: the combination of the student 
X reader interaction and residual error variaxKe accounted for 53 percent of 
the total variance at Grade 4 and 52 percent at Grade 8 (Table 32). 

• There was relatively little systematic difference in leniency among readers 
(see the reader effect in Table 32). 
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Tabic 33 

Pwc«nt«g« of Varianca In Mathaaatica Portfolio 
Scoraa oa a Typical Diaaaaioa that waa 
Attributabla to Varioua Paetora 



Source 


Grade 4 


Grade 8 


Students 


IS 


13 


Students x Pieces 


23 


21 


Readers 


10 


7 


Readers x Students 


5 


8 


Residual 


48 


44 


Total 


100% 


100% 


NOTE: Values may 


not sum to 100% 


because 



of rounding* 

However, unlike writmg, consistent differences among students in total portfolio 
scores (the main effect for students) accounted for only 15 percent of the variance in the 
fourth grade and 13 percent in the eighth grade. In contrast, the main effect of students 
in writing accounted for roughly 30 percent of the variance. Moreover, about one fourth 
of the variance in mathematics scores was due to a student x piece interaction This 
indicates that students received higher scores from both readers on some pieces than on 
other pieces in their portfolios, i.e., the students themselves were not corxsistent in their 
performance level across pieces. The relatively large student x piece interaction 
corresponds to the finding reported earlier that the correlatiorxs between pieces in 
mathematics portfolios were smaller than the correlations between parts in writing 
portfolios. This mear\s that scores on a single piece are a less trustworthy measure of 
student proficiency in mathematics than in writing. Regardless of whether the 
variability in performance across pieces is good news or bad in other respects-and it 
could be either or both-it does indicate that a reliable measure of perfonnaiKe is likely 
to require nnore pieces in mathematics than in writing. 

STRATEGIES FOR IMPROVING SCORE RELIABILITY 

The sections above focused primarily on the reliability of scores assigned to 
specific pieces arni dimensions. In this section, we examine the effects of combining 
scores across pieces arxi dimensions. We also estimate the effects of changing the 
number of pieces in the portfolio ar>d the number of times each piece is graded by a 
different reader. 
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EFFECTS OF AVERAGING SCORES ACROSS PIECES AND DIMENSIONS 

Combining scores pieces and dimensions in mathematics produced only modest 
gairxs in reliability (Table 33). The degree of agreement between readers was still fairly 
low even when scores were summed over all pieces and dimensions. The iiKreases in 
reliability from each type of combining are quite similar to those in writing (Table 18), as 
is the maximum reliability obtained by combining. In mathematics, the maximum 
reliability coefficierto were oiUy .60 and .53 in the fourth and eighth grades, respectively. 
The corresponding correlatiorxs in writing were .49 and ,60. 

Tabl« 33 



Av«rag« Correlation Batman 


Raadara with 


Different Typaa 


of C0Qibining« 


Kath^ 


imatica and Writing 




Mathematics 


Writing 


Type of Score 


Grade 


4 Grade 6 


Grade 4 Grade 6 


No combining --one 


.34 


.37 


.34 .43 


dimension on one piece^ 








One dimension—mean over 


.42 


.36 


.39 .49 


all pieces 








One piece- -mean over all 


.46 


.50 


.45 .56 


dimensions 








Total --mean over all 


.60 


.53 


.49 .60 


dimensions and pieces 









In writing, piece refers to parts, see Chapter 3. 



INCREASING THE RELIABILITY OF TOTAL SCORES BY ADDING PIECES OR 
READERS 

just as in writing, increasing the ntunber of independent readers who evaluate a 
portfolio from 1 to 2 had a noticeable effect on the reliability of total scores regardless of 
the number of pieces within the portfolio that are evaluated. But again, there is not 
much to be gained by having more than 2 readers or more than 5 pieces evaluated. 
However, even under these conditions, the reliability of a student's total score across all 
pieces and dimensions is still only .60. Figure 2 shows the trade-offs among the number 
of independent readers per portfolio and number of pieces evaluated for one of the math 
dimensions. (Similar figures for all of the math dimensions are included in Appendix 
C) 
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Grade 4. P81 




12 3 4 6 



No.ofRiiere 



Figure 2*-E££ecti on reliability of iDcr^Aiiag the auabar of rateri 

or piaeai, grada 4, P81 

CONCLUSIONS 

The results presented in this chapter show that the scores assigned to matheniatics 
portfolios were just as unreliable as the scores assigned to writing portfolios, and the 
low to modest agreement between readers was again the major source of the problem. 
On a given piece, the degree of agreer ient between two readers was orJy slightly better 
than what would occur by charge alone. The mean score assigned to a portfolio on a 
dimension by the first reader had only a low correlation with the mean assigned by the 
second reader. The same was true for the total scores assigned by each reader (i.e., 
across all dimensions arxl pieces). Overall, the mathematics readers were no more 
consistent with each other than were the writing readers, arxi neither group of readers 
provided adequately reliable scores. 

As in writing, there was no apparent value to scoring the portfolios on separate 
criteria (dimensions) because after adjusting for the low reliability of the readers, the 
correlations among the dimensions was near perfect. The sole exception to this trend 
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was that the "PS4 What: Outcomes" dimension. It appeared to measure a somewhat 
different aspect of a student's work than did the other dimensions. 

In writing, students who received a relatively high score on the best part were 
likely to receive a Wgh score on the rest. That did not happen in mathematics. Ir\stead, 
there was a relatively low correlation among the separate pieces (i.e., there was a large 
student x piece interaction). This is the major reason why the total score on a 
mathematics portfolio was not mor€ reliable than the total score on a writing portfolio 
(even thoi^h many more scores were assigned to each mathematics portfolio). 

Finally, increasing fhe number of readers per portfolio from 1 to 2 will increase 
reliability by a noticeable amount, but using more than 2 readers will not help mucK 
Adding pieces helps too, but there are substantially diminishit^ returns in reliability by 
using more than a half dozen pieces. More importantly, there is no operationally 
feasible combination of raters and pieces that will provide an acceptable level of total 
score reliability for a mathematics (or a writing) portfolio. 
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5. THE QUALITY OF AGGREGATE SCORES 

The Vermont portfolio assessment program is designed to provide aggregate 
information, including statewide data and information about smaller aggregates, such as 
supervisory uruor^, districts, and schools.^ The quaUty of such aggregate data depends 
in part on the reliability of individual scores; the low rater reliability described in the 
previous two chapters will decrease the reliability of aggregate statistics. Depending on 
the specific aggregate statistics in question, however, a variety of additional factors come 
into play as well, including: 

1 . Sampling error. In some cases (e.g., eighth-grade mathematics), scores were 
available only for a sample of the state's students.^ Moreover, each year's 
students are in a sense a sample from a larger pool of students flowing through 
the schools over time. This causes some uncertainty in estimates for the entire 
state but is primarily a problem for statistics from smaller groups, such as 
schools or supervisory unions,^ 

2. Clustering. The portfolios of students within schools or classrooms are more 
similar than those of students from different schools. For example, in one sample 
of three schools, a total of 57 eighth-grade mathematics portfolios were scored, 
containing about 80 different lasks; only two of those tasks were conrimon 
between two of the schools. Simple estimates of error assxime that all 
observations (in this instance, students) are independent draws from the 
population, so those estimates of error shoxild be increased to take clustering into 
account. 

^Teachers may use portfolio scores for evaluating individual shidents, but no reporting or 
use of individual shidentj' scores outside of the school have every been planned. 

^ In the 1992-93 school year, all reporting will be based on samples of approximately 1,750 
portfolios per grade in each subject. 

^ The average score for a school is affected by year-to-year changes in the performance of 
successive cohorts of students, independent of any effects of schooling. Research has shown that 
the differences between "good crops" and "bad crops" of students can besiiable. Therefoie, even 
if all students in a school have their portfolios scored, the resulting data do not provide an error- 
free estimace of the level of performance for that school. Rather, it provides an estimate of the 
performance of thai cohori in that school. In ttjhnical terms, this is the reason that we did not 
apply finite-sample cor ections in estimating the error of aggregate scores. 
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3, Biased estimates of proportions. As explained below, unreliability of scoring will 
generally result in too many students obtaining extreme scores, ITus undermines 
a number of aggregate statistics that might otherwise be useful. 

In the following sections, we explore the impact of these factors on the quality of 
both statewide aixl school-level scores. 

STATEWIDE SCORES 

Despite the unreliability with which the work of individual students was scored, 
some statewide statistics, such as average scores on a dimension, were sufficiently 
reliable to report because of the relatively large numbers of portfolios rated i tewide. 
However, other potentially xiseful statistics, such as the proportion of students at a given 
score point, were rendered unusable. Moreover, in judging the reliability of aggregate 
statistics, as in evaluating the reliability of individual portfolio scores, we considered 
only one component of reliability, the consistency of ratings. In this chapter, we make 
no assertions about the reliability of aggregate scores in any broader sense, such as the 
consistency of scores across instances of measurement. For example, we were not able to 
assess the degree to which aggregate scores (such as the ranking of schools) would have 
been affected by substitution of different tasks in st adents' portfolios. 

Average Scores 

Statewide average scores were reasonably precise, as illustrated by fourth-grade 
mathematics composite scores (Table 34), The first column provides the average score 
on each criterioa and the second column indicates how far the margin of error extends 
in each direction. These margins of error are twice the standard error,* This is referred to 
as the margin of error. For example, the average score on "Language of mathematics" 
was t,7 out of a possible 4, and the margin of error extended ,05 in either direction-that 
is, h-om 1,65 to 1,75, We found that these margins of error were in some instances as 
much as twice as large as they would have been with perfectly reliable scoring, but they 
were acceptably small nonetheless. 

*The range extending from the average minus the margin of error to the average plus the 
margin of error is approximately a 95% confidence band. In this case, and elsewhere when 
appropriate, confidence bands were estimated by a school-k?vel jackknife procedure to reflect 
clustering;. 
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Table 34 

Kvmx^gm Pourth-Orada Mathamatiea Conpoaita 
Score a and Uaxgina of Error 



Scoring Criterion 


Average 


+/- Error 


Language of Math 


1.7 


.05 


Math Representations 


2.3 


.05 


Presentation 


2.5 


.06 


Understanding of Task 


2.8 


.04 


How: Procedures 


2.7 


.05 


Why: Decisions 


2.5 


.06 


what: Outcomes 


1.2 


.04 



Average writing scores showed trivially larger margins of error, but again they 
were small enough to be of little consequence (Table 35). 



Tabla 35 

Xvaraga Eighth -<3rada Writing "Raat" Seoraa 
and Margin* of Error 



Scoring Criterion 


Average 


+/- Error 


Purpose 


3.0 


.08 


Organization 


2.8 


.07 


Details 


2.7 


.07 


Voice 


2.8 


.08 


Usage 


2.7 


.06 



Despite the small margins of error, however, statewide averages for 1991-92 still 
had serious limitations. The most important is that some districts arvd schools opted out 
of the proj^ram. In the case of fourth-grade writing, 22 of 246 schools did not contribute 
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portfolios. This represents a relatively small percentage (about 9 percent), but it is not 
random; for example, the largest district in the state (Burlir^ton) vsrithdrew from the 
program before scoring. Accordingly, statewide averages must be interpreted as 
representing only participating schools and districts. Probably less important, portfolios 
were sampled (on both a planned and an ad hoc basis) to compensate for a shortage of 
raters. We have not estimated the likely effects of the non-repr<. jentativeness that might 
have resulted from factors such as these.^ 

Proportions of Students at Each Score Point 

The State Department of Education has been less interested in reporting averages 
than in reporting the proportion of stiKlents reaching each of the foxu* scale points on 
each scoring dimension. There are several reasons for this preference. In theory, the 
proportion of students reaching each score provides more useful diagnostic information 
than does a simple average. Some also believe that avoidir\g averages may help lessen 
the "horse race'* r^ture of comparisons amoi^g schools or districts. 

Unfortunately, the proportion of studenia reaching each score is a more 
problematic statistic under the best of circumstances, and the low rater reliability 
documented in the previous chapters made it unusable in 1991-92. There are two 
problems: large margins of error and bias. 

Margins of Error. In general, reporting of proportions will be more difficult than 
repo rting of averages. The margins of error will often be larger than those of averages. 
Moreover, accurate estimates of the margins of errors for proportions will be difficult to 
obtairt Sampling error and clusterir\g can be addressed straightforwardly in the case of 
proportions, as in the case of averages. However, both measurement error at the level of 
individual students and bias in estimated proportions (discussed below) make it 
difficult to estimate the margin of error in proportions. 

At the statewide level, proportions would have a sizable margin of error even if 
irulividual portfolios were scored with perfect reliability (Table 36).^ In tlte case of 
Presentation, for example, the margin of error would be •f/-4 percentage points. 
Consequently, it would be more reasonable to present the proportion of students scoring 
3 as "43 to 51 percent" or "roughly half," rather than "47 percent." 

' In contrast, in 1992*93, a planned random sample of portfolios was scored in both grades 
and subjects. 

^hese estimates reflect all 1,957 and 1,955 portfolios for which we had \falid scores on 
each of these dimensions, but the margins of error are considerably larger than the conventional 
formula would suggest because of the clustering of scores at the school level. 
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Table 36 

Obs«rv«d Proportion* of Students at Bach Seora Point and 
Margins of Error, Fourth Qrada Mathamatics/ Undaratanding and 
Praaantation (Aaauming Parfaet Ratar Agraamant) 





Understanding 


Present it ion 


Score 


Proportion 


+/- Error 


Proportion 


+/- Error 


1 


2 


1 


9 


2 


2 


17 


3 


39 


3 


3 


81 


3 


47 


4 


4 


1 


0.5 


4 


1 



At the school level, the margin of error for proportions would be very large 
because of the smaller numbers of students. To illustrate this. Table 37 provides the 
margins of error for a proportion of 20 percent, assuming perfect rater reliability, for 
groups of different sizes. Thus, if a school includes 24 fourth-grade students whose 
portfolios are scored (the median fourth-grade enrollment among schools in Vermont in 
1991-92) and 20 percent of them receive a certain score, the margin of error around the 
estimate of 20 percent extends from 4 percent to 36 percent/ 

Worse yet, the margin of error for differences between schools is larger-often by a 
factor of nearly 1 ,5-than the margin of error for estimating the proportions within a 
sir^le school (This is explained further in the following section on school-level scores,) 
Thus, comparisons of proportions among small schools (or other small groups) based on 
a single year of scores is simply not practical 



These are simple random sampling estimates of twice the standard error of a proportion. 
In part because of the small size of many Vermont schools, we assumed no clustering within 
schools. 
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Tabls 37 

Kargins of Error for Proportions of 20 Psrcsnt, 
by Htiabttr of Students (Xsaumlng Perfect Rater Agreement) 



Number of students 


+/- Error (Percentage 
Points) 


15 


21 


24* 


16 


30 


15 


45 


12 


60 


10 


100 


8 



^Median fourth grade enrollment for Vermont schools 
in 1991-9;i 

Bias. When scores are tmreliable, they tend to spread out more than they would if 
scoring were reliable.^ This leads to biased estimates of the proportion of students 
achieving each score. Too many stxidents receive very high and low scores (in this case, 
1 and 4), ind too few receive scores near the middle (2 and 3). 

In the 1991-92 portfolio program, the unreliability of ratings of individual 
portfolios was sufficient to cause serious bias in the proportioi^ reaching each score, 
even at the statewide level. We estimated "true" proportiorw for two criteria in fourth- 
grade mathematics, "tmderstatKlin^' and "presentation/*^ In the case of Understanding, 
we estimated that the true proportion of students scoring either 1 or 4 was essentially 
zero, as opposed to the ob.-i^rved 2 percent and 1 percent (Table 38). The more 
substantial bias, however, was in the scores of 2. We estimated that the true proportion 
of students obtaining a score of 2 was about 8 percent, roughly half the 17 percent 
observed in the data. In the case of Presentation, the estiniated true proportion at a score 

^ This assumes that true scores become less frequent toward the extremes of the scale. 

^ These estimates are based on the assumption thai the unobserved latent scores have a 
bivariate normal distribution. Measurement error was estimated by method of moments-I.e., by 
computing the inter-rater correlation for latent scores necessary to produce the observed 
correlation for the inter-rater data. Monte Carlo methods were used to estimate the relationship 
between these two correlations (40X)C0 draws per point). This procedure introduced some error 
into the estimates but that error is small compared with the sampling error in estimating the 
inter-rater correlations for observed scores. 
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of 1 was 3 percent, rather than the 9 percent observed, and the true proportion at a score 
of 4 was zero rather than 4 percent. Estimates of true percentages, however, rest on 
assumptions that are somewhat risky, particularly when measurement error is as large 
as it was this past year in the Vermont program. Accordingly, we recommended against 
reporting either observed or estimated true proportions for 1992, and the State 
Department of Education followed that recommendation. 



Table 38 

Observed and Entiaated True Proportione of Studente at Each 
Score Point/ Fourth Orade Math, Ukideretanding and Presentation 





Understanding 


Presentation 


Score 


Observed 


True 


Observed 


True 


1 


2 


0 


9 


3 


2 


17 


B 


39 


45 


3 


81 


92 


47 


51 


4 


1 


0 


4 


0 



SCHOOL-LEVEL SCORES 

Scores (such as averages) for small groups are unreliable even when raters show 
perfect agreen^ent on scores for students simply becatise of the small number of students 
contributing to each aggregate score. In the case of the Vermont portfolio program, this 
general problem was exacerbated by unreliable scoring for individual students^ but the 
particularly small size of many Vermont schools woxild result in large margins of errors 
for average scores even if ratings were completely reliable. 

The following statistics, drawn from enrollment data for 1991*92 provided by the 
State Department of Education illustrate the severity of the small-school problem in 
Vermont: 

Number of schools with 4th-grade students: 246. 
Number of schools from which we had 4th-grade writing portfolio 
scores: 224. 

ERIC 
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Minimum number of fourth graders enrolled: 1. 
Median number of fourth graders enrolled: 24. 
Mean number of fourth graders enrolled: 33. 
Maximum number of fourth graders enrolled: 170. 

As noted above, the enrollments in many of these schools are small enough that 
proportions of students reachii^ each score are too unreliable to use. Even simple 
averages, however, are unreliable in the smaUer of these schools, and they would not be 
much improved even if rater reUability became perfect. To illustrate this. Table 39 
presents confidence intervals around means from three actual (but unnamed) schools 
chosen on the basis of enrollments. The table lists the number of portfolios scored in 
writing, the mean "Purpose" score, and three confidence intervals. The first confidence 
interval reflects the actual reUabUity of scoring reported above. The second confidence 
interval is what would obtain if the reliability coefficient (Spearman's rho) were 
Increased to .60, which is probably a reasonable target for the next year or two. The final 
column shows the confidence interval for perfectly reliable scoring; this is unattainable 
for subjective scoring but provides a best-case view of the reUabiUty of school means.^0 
Thus, for example, in School One the 95 percent confidence interval extends 0.4 to either 
side of the mean score of 3.3, given the observed reliability of rating-that is, from 2.9 to 
3.5 . Even perfectly reUable scoring (r=l .0) would only shrink the confidence interval 
modestly, from 0.4 to 0.3. 

These are the confidence intervals one would use if one were drawing inferences 
only about the average score in a single school. If the issue is differences bet ween scores, 
one would need to use a Urger ("simultaneous ") confidence interval. These larger 
confidence intervals for the same three schools are shown in Table 40. 



Note that some changes in the confidence intervals as reliabUity is increased do not 
appear in Tables 39 and 40 because of rounding. 

" Tliese assume that the second school in each comparison is idenHcal in size and in the 
internal distnbution of scores. 
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Table 39 

Confidence Bande for Purpoee* Grade 4 Beet Piece 



School 


Numbor 
of 

Students 


Mean 
Score 


+/- error/ 
Observed 


+/- error, 
r = .60 


+/- error, 
r = 1.0 


One 


14 


3.3 


0.4 


0.3 


0.3 


Two 


31 


2.7 


0.2 


0.2 


0.1 


Three 


53 


3.1 


0.2 


0.1 


0 . 1 



Table 40 



SiJBultan 


eoue Confidence 


Bande 


for Purpoee/ 


Grade 


4 Beet 


Piece 


School 


Nuniber of 


Mean 


+/- error, 


+ /- 


error, 


+/- error, 




Students 


Score 


Observed 


r 


= .60 


r = :.o 


One 


14 


3.3 


0.6 




0.5 


0.4 


Two 


31 


2.7 


0.3 




0.3 


0.2 


Three 


53 


3.1 


0.3 




0.2 


0.1 



Thus, for example, in the case of School One, the observed mean is 3.3, but the mean 
from a a second school of similar size would have to reach 3.9 for us to have confidence 
that it really better. Even with a reliability of .60, which would be a large improvement 
from this past year, a second score would have to be at least 3.8 for us to be confident 
that it is really higher. With perfect reliability of ratings, the second score would have to 
be at least 3.7. Recall that more than a fourth of Vermont's schools are at least as small as 
School One. School Two is considerably larger-recall that sixty percent of Vermont's 
fourth grades are as small as School Two's-but even with moderate rater reliability, we 
could be confident that a second mean is different from School Two's average only if it 
was at least 3.0, compared to School Two's mean of 2.7. 

The differences in averages that are needed to be confident that two schools really 
differ at all are quitp large relative to the observed difference? among schools. In 1991- 
92, we found the following distribution of school means on Purpose for fourth grade 
best pieces: 
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1.8 Mixumum 
2.8 25th Percentile 
3.0 Median, Mean 
3.2 75th Percentile 
4.0 Maximum 

Thus School Two in Tables 39 and 40, with a mean of 2.7, scored below the 25th 
percentile last year. To be precise, it scored at the 20th percentile of the 224 schools for 
which we had fourth grade writing portfolio scores. Yet only schools that are average or 
above average can be considered with any coi\fiderKe to have scored higher than School 
Two.^2 

CONCLUSIONS 

These results show that the urueliability with which the portfolios of individual 
students were rated seriously limits the uses to which the 1991-92 portfolio data can be 
put. Statewide averages had reasoruibly small margirxs of error, but statewide estimates 
of the proportion of students reaching each score point were both biased and unreliable 
and carmot be used. At the level of individual schools, even averages had such large 
margins of error that only very large differences between schools were reliable. 

It is important to realize, however, that attairung higher levels of rater agreement 
will not solve all of these problems. Because of the small size of many Vermont schools, 
many comparisons of average scores among them-if based only on a single year's scores 
on a sii\gle assessment-would be unreliable even if individual portfolios were rated 
with perfect agreement, Comparisorxs of averages among the smaller districts and 
Supervisory Unior\s will be similarly untrustworthy. Moreover, because margins of 
error are poorly ur¥ierstood, publication of those averages could lead people to make 
unwarranted oonclusions-for example, to conclude that chance differences in scores are 
real Finally, because the reliability of portfolio scoring will likely improve only 
gradually, estimates of the proportions of students reachir^ each score point will 
probably remain problematic. 



This is an oversimplification, because the comparisons between School Two and other 
schools depend on the size and distributions within the other schools as well. 
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6. VALIDITY 



Rater reliability is a necessary but insufficient basis for judging an assessment to 
be successful as a measurement tool The ultimate question is the validity of the 
assessment: evidence that the data it yields support the inferences about student 
performance that people base upon it. Unreliable scoring imdermines validity, but 
reliable scoring does not guarantee it. 

It has become common to include under the rubric of "validity" evidence of other 
effects, such as effects on instruction or on the equity of educational services. These 
effects are often labeled "consequential validity" or "systemic validity," In this report, 
however, we use "validity" in its traditional and narrower sense, that is, to refer to the 
quality of the data produced by the assessment. We do not mean to downplay the 
importance of other consequences of an assessment. On the contrary, such other effects 
are a primary rationale for the Vermont program arvi are a major focus of our 
evaluation, (For example, effects on the mathematics curriculum are described in 
Chapter 2.) It is simply clearer to use other terminology to refer to those effects and to 
reserve "validity" for a discussion of the mearungfulness and interpretability of the 
assessments data. 

In 1991-92, the question of the validity of portfolio scores was largely mooted by 
the very low rater reliability. However, we explored evidence of what the validity might 
have been if reliability had been higher. Even though estimates of how scores might 
have behaved in the abserKe of error are risky, particularly when the error is as large as 
It was in the Vermont portfolio system, the effort was instructive. It provided some 
initial hints about the validity of the Vermont portfolio scores. It also illustrated a 
number of impediments to effective validation, not only of the Vermont program 
specifically, but also of many performance assessment systems, 

CRITERIA FOR VALIDATING THE VERMONT PROGRAM 

A wide variety of evideruie can be adduced to test the validity of an assessment. 

In 1991-92, we focused both on evidence from the scores themselves and evidence from 

our investigation of the implementation of the program. 

One criterion we examined is generalizability of performance: the consistency of 

students' performance over alternative measures of the same construct or achievement 

domaia The validity of inferences about students* performance in a given domain 
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would be undermined by evidence that their performance does not generalize well 
across measures. For example, suppose that one had two alternative tests of algebra, 
judged to be roughly equivalent in their diffioxlty and their representativeness of the 
domain of algebra. If st'udents' performance on one of the tests was inconsistent with 
performance on the other-that is, if their performance did not generalize across the two 
tests-one could not consider either one a valid basis for judging students' ability in 
algebra, because they would suggest different conclusions about which sttjdents had 
mastered the subject. Generalizability is a matter of reliability as well as validity and is 
sometimes called "score reliability." 

We also looked at "convergent" and "divergent" evidence. These cumbersome 
terms refer to a fairly simple notion: scores on a test should correlate more highly with 
measures of highly related constructs than with measures of less related constructs. For 
example, proficiency in calculus should be more highly related to proficiency in other 
aspects of mathematics-say, trigonometry-than with vocabulary. Therefore, if scores 
on a test of calctilus failed to correlate more highly with trigonometry scores than with 
vocabulary scores, one would have good reason to doubt the validity of the calculus 
scores. 

For comparisons of this sort, we used scores from the state's imiform assessments 
of mathematics and writing. The uniform test of mathematics was a matrix-sampled test 
that included both multiple^ihoice and open-ended i^ems.^ However, the open-ended 
items were not scored, and the scores used by the state were based on the 30 multiple 
choice items administered to each student. In our study of validity, we similarly used 
only the multiple-choice items. The writing uniform test was a single essay written in 
response to a sii\gle prompt used throughout the state. The essays were scored by 
employees of the state's testing contractor rather than by Vermont teachers, but the 
raters used the same scoring criteria and rubrics that were used to score the writing 
portfolios. We also collected mid-year and final mathematics grades and all 
standardized test scores for students in a stratified random sample of Vermont schools. 

Interview and questionnaire data also provided evidence pertaining to validity. 
For example, as noted in Chapter 2, we fourxl evidence pertaining to variations in the 
implementation of the program that might bear on the validity of scores. We also 
obtained useful feedback from portfolio raters. 

matrix sampled test is one in which each student receives only a sample of the total 
pool of lest items. 
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BARRIERS TO VALIDATION 

The barriers to validation we encountered were numerous. Some were 
idiosyncratic aspects of the Vermont program or of the context in which it operates, but 
others were factors that are likely to affect a variety of performance assessment systems. 

As noted above, the first obstacle to validation was the low level of rater reliability 
we fotind; an unreliable measure cannot be valid. Moreover, the reliability of ratings 
was so low that it clouds our estimates of what relatiorxships among scores would have 
been if rating had been better. There are techniques for estimating what relationfhips 
would be found if rater reliability had been high-the disattenuation methods used in 
Chapters 3 and 4**but when error is as large as it was in the case of the portfolio scores, 
the resulting estimates are uncertain. 

Our sample schools used a variety of different standardized tests and employed a 
wide variety of grading methods and standards. Accordingly, we planned to conduct 
our analysis of both grades and standardized test scores within schools and then to pool 
the results across schools. The very small erut)llments in many Vermont schools would 
have made that difficult at best, and the low reliability of scores would have exacerbated 
the problem of small nximbers. The decision at the end of the year to score only a 
sample of mathematics portfolios from ea :h school, necessitated by an insufficient 
number of raters, made most within-school samples too small for the planned analyses. 

Of more general importance than these concrete problems, however, was the 
insufficiently clear definition of the domain of mathematics that the portfolio assessment 
was supposed to tap and the lack of a clear rv}tionof the relationships that should obtain 
between that domain and others, such as more traditionally assessed aspects of 
mathematics and aspects of verbal fluency. (Recall that several of the seven 
mathematics criteria pertain to communicatioa) This is a problem that will plague 
many performance assessment programs. One rationale for performarKe assessments is 
that they will measure different aspects of competence than do traditional tests. 
Moreover, many are intended to bridge more lhan one traditional domain. (The 
Vermont mathematics portfolio program, with its emphasis on written commtmication, 
is a clear example.) Thus, up to a point, a moderate correlation between mathematics 
portfolio scores and scores on a multiple-choice test might be construed as better news 
than a very high correiatioa A very high correlation might signify that the portfolios 
were not providing much ir\formation beyond that available from the multiple-choice 
test, while a moderate correlation might indicate that the portfolios were successfully 
tapping other aspects of proficiency in mathematics. A substantially lower correlation, 
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however, might be damning; it might indicate that the portfolio scores are too lit-oivily 
influenced by things that are no^ gf?;iruine, (For example, many teachers have expressed 
concerns that the emphasis on written communication in the mathematics portfolio 
program, whatever its instructional benefits, might be undermining the validity of the 
scores for fourth-grade students with relatively weak proficiency in verbal expression 
and writing.) 

Thus, there ib as yet no firm basis for deciding what would constitute good or bad 
news. Clear evidence of validity will require more clarity about the patterns of 
relationships that portfolio scores should show (particularly outside of the area of 
writing), and it will likely also require an expanded range of measures that can be used 
for comparison. 

EVIDENCE FROM THE WRITING PORTFOLIO SCORES 

As noted in Chapter 3, scores on the two parts of the writing portfolios were quite 
consistent, after removing the effects of rater uru-eliability. Indeed, after disattenuating 
to remove the effects of imreliable rating, the correlatiorxs between the two parts of the 
portfolio were higher than some other research would have predicted. 

Some readers might construe that consistency between scores on the two parts as 
evidence of generalizability of performance. There are several reasons to be cautious, 
however The portfolio program may have produced a limited and non-represenitative 
sample of the domain of writing. Even if the program as a whole sampled reasonably 
well from the domain, it is quite possible that the portfolios of individtjal students did 
not. (Recall that in mathematics, we found evidence of marked between-school 
differences in task selection.) Moreover, the extremely low level of rater reliability 
makes the disattenuation suspect. It is possible that if raters had scored reliably, the 
correlations shown by their scores would have been different from our disattenuated 
estimates. 

For these reasons, we looked at the state's imiform test of writing for additional 
convergent evidence of validity. When we compared scores on the writing portfolio to 
scores on the writing uiuform test, we obtained lower correlatior\s than we found 
between the parts of the portfolio. The raw correlatiorxs were very low, as the low rater 
reliability c / the portfolio scores preordained, In grade 8, for example, the average 
correlation (across dimensions) between the best piece and the rest of the portfolio was 
.37; the average conrelatior\s between the uniform test and the best and rest were .31 and 
,28, respectively. After disattenuating for rater imre liability, however, the correlations 
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between scores on the portfolio and the uiuform lest were muderate, ranging from .47 to 
.59, compared to .80 or higher for the correlation between the best piece and rest (Table 
41). The rest scores showed a bit higher correlations with the uniform tt3t than did the 
best piece. It is possible that the larger number of pieces of work entering into the rest 
scores made those scores a bit more robust. 



Table 41 

Oi0att«nuated Corralationa Batwaan Writing 
Piacaa* Portfolio And Uniform Taat 









Portfolio 










(rest) 


Portfolio 






Portfolio: 


versus 


(best) 






best piece 


uniform 


versus 






vs. rest 


test 


uniform test 


Grade 


4 


»80 


-59 


• 47 


Grade 


8 


.86 


.58 


.52 


Note : 


Averages across 


dimensions 


of correlations 



calculated within dimensions. 

These disattenuated correIatior\s are reasonable, in the light of other research.^ 
They suggest that although much of the variation in ratings is error, raters are to some 
degree recognizing differences in the quality of writing. However, only more reliable 
scoring will permit us to test whether this inference based on disattenuation is correct. 

For divergent evidence, we also compared writing portfolio scores to scores on the 
mathematics uruform test. Because we considered only scores from multiple-choice 
items and ignored items that required writing, one would expect that the writing 
portfolio scores would show a lower correlation with scores on the math uniform test 
than with scores on the writing uniform test. This was not the case. In the fourth grade, 
writing portfolio scores showed nearly identical correlations with the two uniform tests 
(Table 42). In the eighth grade, correlatiorxs with the writing uniform test were higher 
than those with the mathematics test, but only marginally. These correlations, however, 
are ambiguous because of the limited scope of the writing uruform assessment. That is, 

^The uniform test was a single prompt, so the correlations between the best piece and the 
uniform test are analogous to the correlations between single essays reported in other research, 
(See Dunbar, Koretz, and Hoover 1991.) 
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the fact that the writing iiniform assessment comprised only a single prompt would tend 
to depress the correlations between that test and the writing portfolios. 

Tabltt 42 

DiflattMuated Corralationa Mtwa^n Writing Portfolio Scores 
And Uniform Taat Seoras In Writing And Math 









Writing 


Math . 








uniform test 


uniform test 


Grade 


4 


Best 


.47 


,50 






Rest 


.59 


.61 


Grade 


8 


Best 


.52 


.43 






Rest 


.58 


.52 



EVIDENCE FROM THE MATHEMATICS PORTFOLIO SCORES 

Correlations between scores on the mathematics portfolio and the mathematics 
uniform test are also difficult to interpret because of both the severity of the 
disattenuation for rater error and the lack of a clear expectation for the relationships that 
should obtain between the portfolio scoring dimensions and other aspects of 
performance in nuthematics. Nonetheless, there is little in the correlations found in 
1991-92 to generate confidence in the portfolio scores. 

As noted in Chapter 4, scores on the mathematics pieces within a portfolio were 
less consistent with each other than were scores on the two parts of the writing portfolio. 
The average raw correlations between the pieces varied from one dimension to another, 
but all were very low; 11 of the 14 correlations were less than .20. Even after 
disattenuating for rater unreliability, the average correlations between pieces remained 
low: the overall average (across dimensions) was .38 in fourth grade (Table 43, left-hand 
column). Eighth grade results were similar; the overall average (across dimensions) was 
.31. 

Accordingly, we compared composite scores from the portfolios, rather than 
piece-level scores, to scores on the mathematics uniform test. Given the low correlations 
among piece-level scores, the composite score should be more reliable and should show 
higher correlations with other measures of performance in mathematics. 

Even after disattenuatioa the correlations between mathematics portfolio 
composite scores and the math uniform test were typically quite low, averaging about 
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.32 C able 43, right-hand column). These correlations showed a different pattern across 
dimensiorxs than did the correlatiorxs among the portfolio pieces themselves (Table 43, 
left-hand column), but the average correlation is lower. There is also little variation 
across dimer^ions in the correlations tetween portfolio composites and the uniform test, 
even though one might expect the dimensions to vary in their relationship to the 
knowledge and skills tapped by a traditional mathematics test 



Avarag* DiMttanuatad Comlationa Batvaan Mathamatica 
Portfolio Piacaa And Math Unifom Taat Scoraa, Grada 4 



Dimension 


Portfolio 
pieces with 
each other 


Portfolio 
composite with 
viniform test 


Language 


0 »21 


042 


Pepresentat io;is 


0.15 


0.31 


Presentation 


0.54 


0.36 


Understanding of task 


0.39 


0 .41 


How: Procedures 


0.42 


0.33 


Why: Decisions 


0.57 


0.32 


What: Outcomes^ 


(0.38) 


(0.08) 


Mean 


0.38 


0.32 



^Correlations on this dimension are of questionable 
meaning because scores showed almost no variation. It has 
little effect on the mean, however; the disattenuated mean 
without this dimension is .35. 



More ground for pessimism appeared when mathematics portfolio scores were 
compared to the uniform tests in both writing and mathematics. One would expect 
math portfolio scores to correlate more strongly with the mathematics uniform test than 
with the writing uniform test, both for substantive reasor\s and because of the limited 
scope of the writing test. Further, one might expect the correlations to vary among 
dimensions in predictable ways. In the following tables, we present correlations 
involving two dimensions each in writing and mathematics. In mathematics, we 
selected Presentation because it was the most reliable of the communications dimensions 
and Procedures because it seemed most related to the skills that would be needed to 
solve problems on the mathematics uniform test. In writing, we selected Usage, 
Grammar, and Mechanics because it was the most reliable scoring dimension and 
because it entails a discrete set of skills (such as punctuation and parallel use of tense) 
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that should be largely imrelated to scores on the mathematics portfolio. We also 
selected Orgaruzation because we reasoned that it might be more similar to some of the 
skills needed for the math portfolio; poor organization of presentations would lower 
scores on the mathematics portfolio as well as on the writing test. 

Thus, we had a munber of expectations in examining the correlatiorxs between 
scores on the math portfolios and the two uniform tests. We expected math portfolio 
scores to be more highly correlated to scores on the math uniform test than on the 
writing uniform test. We expected math portfolio scores to have the lowest correlation 
with Usage. We expected Procedures to have a higher correlation than Presentation 
with mathematics uniform test scores, and we expected Presentation to show higher 
correlations than Procedures with writing uniform test scores. 

In the main, these expectations were rK>t borne out. In fourth grade, mathematics 
portfolio scores (averaged across dimensions) showed nearly identical correlations with 
writing Orgaruzatioiv writii^ Usage, and the mathematics imiform test (Table 44). In 
eighth grade, mathematics portfolio scores showed trivially higher correlatiorxs vsrith the 
writing uiuform test than with the mathematics uniform test. 



Tabl« 44 

AvKftgc Disattsnuatcd Corr«latien« Mtwaan Math Portfolio 
Scovmm And Ur^ifona Tact (UT) Scons In Writing And Math 





Writing UT: 


Writing UT: 


Math 




organization 


Usage 


UT 


Grade 4 


.33 


.33 


.35 


Grade 8 


.35 


.38 


.31 



In general these correlations were similar across the seven mathematics portfolio 
dimensions, bvl there were a few dimensions that differed. There was some limited 
concordance with our expectatiorxs in eighth grade. The math portfolio Procedures 
scores did in fact correlate substantially more strongly with math uniform test scores 
than did the Presentation scores (Table 45). However, any optimism fostered by that 
pattern is tempered by the fact that the math portfolio Procedures scores correlated 
nearly as well with both of the writing dimensions as with math uruform test scores. 
(Recall that we expected the correlation between math Procedures and writing Usage to 
be particularly low.) 
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DiMttanuatttd Cor»lation« B«twa«n Math Portfolio Seor«* 
And UT Seor«« In Waiting And Math/ Grad* 8, By Dinanaion 



Math dimension 



writing UT: 
Organization 



writing UT: 
Usage 



Math 
UT 



Presentation 
Procedures 



.32 
.38 



.37 
.39 



.19 
..4 2 



EVroENCE ABOUT PROGRAM IMPLEMENTATION 

Our data on t)>e implementation of the program also cast some doubt on the 
validity of scores when used for cer^^-m purposes. As noted in Chapter 2, teachers 
report wide variations in their implementation of the program, and some of the 
variations they report could ha ve a substantial impact on the mearung of scores. For 
example, differences in rules about revision— how much revision is allowed, how much 
guidance is provided for revision, and what help is allowed from parents and others- 
could substantially influence scores. Such variations would undermine the validity of 
comparisons between classes or schools that had substantially different practices. Other 
variations in implementation that could threaten validity would be differences in the 
extent of preparation for tasks (and, conversely, their degree of novelty) and differences 
in the presentation of tasks. 

The considerable differences in task assignments we found in our qualitative 
review of mathematics portfolios could similarly undermine the validity of compa risons 
among schools. Some tasks afford more opporturuty than others to display the scored 
competencies. Similarly, as some raters have pointed out, one can increase the 
probability that a student will score well on the portfolio dimensions by assigrung tasks 
that are relatively easy. Even when tasks are nominally the same, teachers can assign 
easier or more difficult variants.*^ 

CONCLUSIONS 

The evidence presented here, although exploratory and tentative, suggests that the 
validity of Vermont portfolio scores in mathematics may be questionable. It also 

^Ratere in tho 1993 scoring discussed this problem in some detail and provided concrete 
examples. 
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illustrates the difficulty of validating assessments of this sort and suggests that 
researchers will need to cast their nets broadly to get an adequate view of vahdity. 

Our data collection in the 1992-93 years will add additional information relevant 
to validity. For example, Vermont added a "portfolio-like" task to the mathematics 
iirdform test in the spring of 1993, which will provide another useful comparison to 
portfolio scores. At our request, a subsample of writing portfolios were scored by an 
alternative method in which scores for each piece in the "rest" were separately recorded, 
in addition to the "rest" score. We observed criterion sessior\s (in which raters scored 
benchmark pieces and then debated their scores) and recorded information on the bases 
or raters* disagreements. We also obtair^d feedback from a substantial number of 
mathematics raters about factors relevant to validity. 

Nonetheless, further expansion of methods and measures will be needed to get a 
solid understandii\g of validity. In particular, validation of scores from the Vermont 
program-and similar programs-will require clarification of the domains that the 
assessments are designed to measure. This is likely to entail both dearer conceptual 
definitions and more explicit delineation of the types of tasks and performances that are 
expected. 
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7. IMPLICATIONS 



The Vermont program has some unusual features, and some of the findings 
reported here reflect its idiosyncrasies. However^ our findings also have important 
implications for performance-assessment programs more generally and for the design of 
research to evaluate the quality and effects of those programs. 

Vermont's program differs from many current large-scale performance- 
assessment programs in its reliance oa portfolios. While most large-scale programs rely 
primarily on standardized products, such as students' performances on st indardized 
tasks or essays written in response to standardized prompts, Vermont uses standardized 
performance assessments orJy as components of its "Toniform" tests. The portfolios are 
unstaiKlardized. The Vermont approach is also a typically "bottom-up." For example, 
largely volimteer committees of teachers (rather than State Department experts or 
outside contractors) have much of the responsibility for designing rubrics and 
establishing guidelines for the form and content of portfolios. 

As important as these characteristics may be, they should not obscure the many 
similarities between the Vermont assessment and other programs or the implications of 
the Vermont experience for these other efforts* For example, the Vermont program 
shares with many programs the dual, fundamental goals of measuring student 
performance and spurring improvements in educational practice. The sf>ecific types of 
instructional change the Vermont program is intended to spark (such as more extensive 
writing throughout the curriculum aivi more emphasis on problem solving and 
communication in mathematics) are also among the primary goals of many other reform 
efforts across the nation. Moreover, many current proposals call for assessment systems 
that are similar to the Vermont program. For example, portfolios and other 
unstarxlardized products are central to the proposals of the New Standards Project. 

For these reasons, the Vermont experience has substantial implications for the 
performance-assessment movement nationwide. The results described here can help set 
expectations for other programs and provide guidance for their design In this chapter, 
we discuss four issues of general importance to performance assessment that are 
illuminated by the Vermont experience: expectations regarding the quality of 
measurement; expectations regarJing the impact of assessment on educational practice; 
the fundamental tension between Ihe goals of educational improvement and 
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measurement quality that motivate this and other performance assessment programs; 
and requirements for program evaluatioa 

EXPECTATIONS FOR QUALITY OF MEASUREMENT 

In 1991-92, the Vermont program was largely unsuccessful in providing high- 
quality information about student achievement. The reliability of scoring was so low 
that it precluded most of the intended uses of the portfolio scores. Moreover, both 
patterns in the scores themselves and variations in the program's implementation raise 
doubts about whether the scores would have provided a valid basis for certain 
conclusions— among them, comparisons across schools or other groups— even if the 
scoring had been more reliable. 

A key question for policy is why the assessment data were so weak. For example 
members of the Vermont State Board of Education wanted to Imow how much of the 
xmreliability of scoring is a consequence of using portfolios and how much improvement 
in reliability could realistically be expected if the program was improved but continued 
to rely on portfolios. Similarly, observers outside of Vermont want to kmw how much 
the problems doctimented here can be attributed to factors that will affect their own 
programs. 

Our view is that the problems encountered in Vennont should serve as a signal to 
set modest expectations for the quality of data from innovative performance 
assessments, particularly over the short- and moderate term. Although the problems in 
Vermont stem partly from idiosyncratic factors, they also appear to reflect factors 
relevant to many performance-assessment programs. 

Our observations of the program suggest at least three possible causes of the 
unreliability of scoring— problems with the scoring rubrics, insufficient training, and the 
lack of standardization of tasks-but we lack the data at this dme to disentangle their 
relative contributions. Although all three have aspects that are unique to Vermont, it is 
likely nonetheless that similar problems will arise in other programs, particularly those 
that rely on portfolios or other types of unstandardized performance assessments. For 
example, reliance on non-standardized tasks, however desirable for possible effects on 
instruction^ will often severely complicate efforts to devise reliable scoring rubrics and 
methods, particularly in subjects (such as mathematics and science) in which tasks are 
likely to vary greatly.^ Similarly, the taik of training large ntimbers of teachers to score 

* We are aware of one portfolio program in which in lerrater agreement was far higher 
than in Vermont. A recent portfoiio assessment of writing in Pittsburgh achieved in terra ter 
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reliably will generally be difficult in most programs and v/ill be especially hard in 
programs that require teachers to grade disparate products using methods not tailored 
to any specific tasks. 

Similarly, the problems of validity suggested by our data most likely stem at least 
in part from factors common to many assessment programs. For example, our 
generalizability analysis of mathematics portfolio scores showed large task-to-task 
variations in the scores of individual students that would threaten the validity of 
inferences about student performance based on a small number of tasks. Far from being 
unique, this limited generalizability of performance across complex tasks within a 
subject area is the norm in the research on performarure assessments. (See, for example, 
EHmbar, Koretz, and Hoover, 1991, and Shavelson, Baxter, and Gao, 1993.) The large 
variability we found in teachers' implementation of the Vermont program is also likely 
to be mirrored in othe:- programs that attempt to integrate assessment into teacher- 
directed instruction, and it will pose potential threats to validity in those programs as 
well. 

Whatevei its causes, the effects of unreliable scoring discussed in the preceding 
chapters are not unique to Vermont. The appropriate uses of scores will be limited 
whenever sinular problems of reliability arise. Unreliable scoring will of course always 
urxlermine the utility of scores for making decisions about individual students. 
Moreover, as the results above illustrate, it will also threaten inferences about 
^gg^g^*^^- For example, unreliability of scoring will generally bias the distribution of 
scores, causing too many students to score at the extremes and too few near the middle. 
Thus, estimates of the proportion of students reaching various points on the scale will be 
misleading overall, and comparisons between groups (schools, districts, demographic 
groups, or whatever) that differ substantially in their average scores will be error-prone. 
Similarly, although complex analytical scoring systems may be beneficial as incentives 
for instructional change, they will not yield meaningful data if the various scoring 
dimer\sions cannot be distinguished reliably by raters. It is true that these problems 
were extreme in Vermont in the 1991-92 program, but they would remain substantial 
even with considerable improvements in reliability. For example, even if reliability at 

correlations above .70 (LeMahieu, 1992). We suspect that one reason that Pittsburgh attair^ed 
reliability so much higher khan that of the Vermont writing portfolio program is that in the 
Pittsburgh program, portfolios were scored by a relatively small group of people who had long 
involvement in the program . We are not aware, however, of any large-scale program that has 
achieved comparably high levels of agreement in mathematics or science portfolio assessments in 
which the contents of portfolios are as unregulated as they have been in Vermont. 
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the level of scoring dimension.^ was increased to .70«-which would represent a very large 
unprovement--fuUy half of the variance in students' scores would still be error^ and that 
much error would substantially bias the proportion of stiident reaching each score. 

EXPECTATIONS FOR IMPACT ON EDUCATIONAL PRACTICE 

At least in mathematics, the Vermont assessment program appeared to be more 
successful ii; 1991-92 as an educational intervention than as a measurement program. 
Principals and teachers agreed that the program provided a powerful impetus for 
change in instruction^ and the reported changes, such as an increased emphasis on 
problem solvir>g, appeared to be largely consonant with the goals of the program. The 
fact that so many schools opted to expand their use of portfolios beyond the fourth and 
eighth grades despite the large burden it imposL\l is a telling measure of its perceived 
positive effects on instructioa 

Many observers— we among them— see these preliminary findings as grounds for 
optimism about the potential effects of innovative assessments on instructional quality. 
There are, however, reasons to temper that optimism. The evidence to date about the 
effects of the program is both limited and mixed, and the Vermont experience 
underscores how difficult it is to obtain desired outcomes. 

It is important to reiterate some of the most important limitations of the data 
reported here. The information we report on the effects of the program reflect primarily 
self-reports: interviews with principals and teachers and an anonymous teacher 
questionnaire. Our qualitative analysis of portfolios was limited in scope and not 
necessarily representative of the state as a whole. Our direct observation of classrooms 
was extensive in terms of sampling but very limited in duration and depth; although it 
provided useful examples and clarified and supplemented some of the responses we 
obtained h*om educators, it was not sufficient to provide a systematic check on the 
accuracy of self-reports. Moreover, the ultimate test of instructional improvement is 
enhanced learning. The data we had h-om the 1991-92 implementation offer no direct 
measure of effects on student learning, 

If one accepts the reports of teachers and principals as an indication of positive 
effects on instruction, there are still reasons to be cautious about the extent and 
pervasiveness of that impact. One reason is the patterns shown by the portfolio scores 
themselves. The unreliability of scoring suggests irteonsistent interpretation of 
performance goals by the state's teachers, and that in turn raises the prospect of 
inconsistent instructional goals and practices. The apparent lack of independence of 
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most of the scoring dimensions and the relatively minor differeiKes betwreen the best* 
piece and rest scores in writing indicate that on average, teachers would learn no more 
from the many scores assigned to each portfolio than they would from a single score. 
This raises the question of whether teachers are in fact providing students with 
reasonable and consistent feedback on their efforts to meet the perforrnarKe goals 
reflected in the assessment's many dimensiorxs and components. (It is possible, 
however, that requiring scores on multiple dimensior« caused teachers to focus 
instruction on all of them, even if the scores were not a reasonable base for monitoring 
their efforts.) 

In additioa while the Vermont assessment program was apparently a powerful 
method of signaling to teachers what was expected of them, the evideivre suggests that 
its success in this respect was incomplete. Indeed, the Vermont experience argues that 
much more than an assessment is needed to accomplish this goal Our questionnaires 
and other observations show that in mathematics, the Vermont program, including the 
provision of illustrative tasks in the Resource Book and corxsiderable training, had 
apparently substantially altered teaching in the aggregate. However, it had not been 
sufficient to create a consistent understatKling of what corxstitutes appropriate teaching. 
This is not an entirely negative finding; some observers see the spirited debate among 
Vermont teachers about curriculum aiKl instruction that continues even two years after 
the inception of the program to be one of its greatest benefits. Nonetheless, this is one 
more instance in which the Vermont experience suggests moderate expectations. It is 
one thing to communicate to teachers that they should put more emphasis on problem 
solving; it is quite another to communicate effectively what that mearxs and how it can 
*^ accomplished in a way that actually improves students' skills. To do so is likely to 
require a great deal of time and effort. 

This points to yet another reason for caution: the major costs of the progress made 
to date. There has been no accounting of the direct and indirect financial costs of the 
system; indeed, given the extraordinarily decentralized nature of the Vermont 
educational system and of responsibility for this program, it would be difficult to obtain 
one. It is clear, however, that the costs in time, effort, and stress have been large. 
Indeed, the burdens noted in this report represent only the initial stages of a continuing 
and still difficult and costly process of program development. 

Finally, one has to ask here the same question we asked about quality of 
measurement: to what extent do our findings reflect the idiosyncrasies of the Vermont 
program? It is our impression that the answer is again mixed. The nature of the 
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assessment tasks themselves, particularly in mathematics, clearly did to some degree 
signal concretely to teachers what was meant by otherwise abstract goals such as 
'increasing the emphasis on problem-solving." One would expect that this signaling 
function could be served by diverse performance assessment programs quite unlike the 
Vermont program. However, it is our impression that the impact of the program also 
has stemmed in part from aspects of the program that are relatively unusual, albeit 
replicable. For example, the support of teachers and principals appears critical to the 
effective operation of the program^ and our interactions with Vermont educators 
suggests that both the decentralized, bottom-up nature of the program and exter\sive 
and time-consuming efforts by the State may have been critical in building that support. 

TENSIONS BETWEEN THE GOALS OF ASSESSNfENT PROGRAMS 

To what extent can assessment programs be expected to meet the dual goais of 
improving instructioit and providing high-quality information about student 
achievement? Given that both goals are fundamental to the current performance- 
assessment movement, the answer will have v/idespread ramifications for education 
reform. It is not surprising that the Vermont program appeared to be considerably more 
effective in meeting one goal than the other during its first year of its implementation. 
More important are implications of the Vermont experience for the longer-term potential 
of meeting both goals. 

Our view is that the goal of improving instrut m often cortflicts with the goal of 
providing high-quality, valid, and reliable data about student performance. More 
concretely, an assessment program designed primarily to meet the first of these goals 
would likely be quite different from one designed primarily to meet the second. For 
example, standardization of tasks and adnunistrative conditions will generally improve 
the quality (at least, the comparability) of data about student performance, but those 
same attributes are likely to impede the integration of assessment and curriculum and 
may undermine teachers' feelings of ownership and connmitment to the program. For 
programs (such as Vermont's) that have both goals, success will depend on finding a 
workable compromise between the two, deciding, for example, what price in 
measurement quality is acceptable to gain additional leverage on instruction. The 
founders of the Vermont program, imlike many other reformers, openly confronted this 
dilemma at the outset, but experience is beginning to show how difficult it will be to 
resolve it. 
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The tension between the goals of the program became apparent in a variety of 
contexts in Vermont. For example, one consideration that led us (and some teachers) to 
question the validity of comparisons based on portfolio scores was the large variation in 
key aspects of program implementation, such as policies toward the revision of students' 
wjrk. Clearly, such variations can threaten validity. Two students of similar 
competence might produce comparable products when confronted with the same 
constraints but dramatically different products if one is allowed much more time to 
revise or is given more help (by teachers, parents, or other students) in revising. Yet 
some of the variations in instruction that could undermine the validity of comparisorxs 
may be precisely those one wishes to encourage to improve instruction. An effective 
teacher may decide, for example, that less able students need more help than more able 
students, and perhaps more structured directive help, in revising products. The teacher 
may believe, for example, that the more able students are at a point where they should 
learn to work more autonomously in revisii>g their work. This variation in procedures 
may help both groups of students, but it will uixiermine the validity of comparisorxs 
based on the scores by making differerures between the groups appear smaller than they 
really are. Conversely, other differences in revision rules might exaggerate differences 
incompetence. 

The tension between the goals of measurement and instructional improvement 
also arose in developing procedures for scoring portfolios. The goal in Vermont has 
been to involve all teachers in the affected grades in scoring portfolios. This policy 
stems directly from the instructional-improvement goals of the program: training in 
scoring student work is seen as a critical component of training teachers to understand 
the instructional goals of the program. Yet, the more broadly resporxsibility for scoring 
is shared, the more difficult it becomes to provide enough trairung to brii>g scorers to an 
acceptable level of proficiency, and the more likely it becomes that insufficiently 
proficient raters will participate in the scoring process.^ 

2 In response to this conflict in goab, we suggested to the State Department of Education 
that scoring be conducted on two separate tracks. The Department would continue to provide 
training in the scoring process to all teachers and would continue to request that all teachers 
score. However, scores for reporting by the Department would be generated separately at a 
single workshop at which raters would receive additional training and would be monitored. The 
Department followed this suggestion in the 1992-93 school year, and the effects of this change 
and further maturation of the program on the quality of scores will be discussed in a forthcoming 
report. 
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REQUIREMENTS FOR EVALUATION 

Many performance assessment programs, including Vermont's, are intended to 
have pervasive effects throughout the educational system. These programs, however, 
like more traditional forms of test-based accountability, are not self-evaluating. Upward 
trends in scores on the new measures are to be expected and are not sufficient to indicate 
that the goals of the programs are being met. The range of questions that need to be 
investigated to evaluate such a program is illustrated by otir experience in Vermont— 
both by the findings noted above and by the many questions our data do not address. 

Documenting program implementation. The findings described in Chapter 2 
illustrate the importance of documenting, not only typical patterns of implementation, 
but also variations among teachers, schools, and categories of students. This 
information is important for formative purposes-that is, to identify problems that need 
to be addressed as the program matures. Variations in implementation also may have 
important implications for equity. A preliminary investigation of variations in 
principal's responses to our interviews did not reveal striking differeiKres between large 
and small schools or between high-poverty and other schools. Nonetheless, it seems 
likely that in other contexts, variations in program implementation (and quality) may be 
associated with factors such as socioeconomic status or ethrucity.^ As we noted earlier, 
variations in how programs are implemented may also have substantial implications for 
the validity of the assessment results. Information on differences in implementation can 
alert users to this possibility and can be helpful for designing validation studies. 

The Vermont experience also underscores the importance of a far-ranging 
investigation of costs and burdens. Particularly when assessments require substa ntial 
efforts by classroom teachers, it is simply inadequate to tabulate only the direct costs 
borne by states or large districts. The non-financial costs are diffused throughout the 
educational system, and financial costs may be hidden in other budget categories at 
lower levels of the system. (One example of this that we found in Vermont was the 
substantial allocation from districts' substitute budgets to pay for release time for 
training.) Propor^ents of systemic refomns based on f>erformance assessment often 
maintain that some of the burdens imposed by the assessment program should not 

^Vermont has a substantial poor population, but it lacks many of the other social divisions 
that are of concern to other jurisdictions. In addition, the mechanisms by which teachers sort 
themselves among schools may be very different in Vermont (where most districts operate only a 
few schools and are geographically dispersed) than in jurisdictions with large districts and 
geographically concentrated schools. 
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properly be considered purely costs of assessment For example, they argue that some 
of the time teachers spend adapting to the systems should be treated partly as costs of 
professional development or curriculum improvement. The VenxKjnt experience is 
consistent with this view; for example, the large amount of time some teachers devoted 
to finding appropriate tasks can clearly be seen as curriculum improvement as well as a 
cost of assessment. However, this is an argument for more complete investigation of 
costs and burdens, not less. 

Investigating instructional effects. Our experience suggests that more direct 
measures of instructional change would be very valuable. However, they may be 
difficult to obtain. E valuators should ideally obtain baseline measures before the 
program is implemented, and the current political climate-exacerbated as it has been by 
widespread, imrealistic expectations ab>ut the speed with which reforms of this sort can 
be effected-may make it very difficult to put data collection into place before programs 
are fielded. Moreover, in large-scale and geographically dispersed assessment 
programs, the costs of some direct measures may be prohibitive. Other research on 
curriculum-for example, the continuing efforts to develop more sensitive measures of 
curriculum for international studies of achievement-may provide measures that are less 
burdensome than direct observation but that are still useful for evaluating programs of 
this kind. 

Assessing reliability. Estimates of interrater Agreement are necessary but clearly 
insufficient. Evaluators need to explore both the causes and the effects of differences 
among raters. Their causes will often prove more difficult to ascertain in programs that, 
like Vermont's, rely on unstandardized products and general-purpose scoring rubrics. 
For example, our data yielded estimates of the variability of performance across tasks in 
mathematics but did not provide any information about the impact of specific task 
characteristics because tasks are not specifically identified.^ In standardized 
performance assessments, by contrast, aspects of tasks can be systematically varied. 
Unstandardized assessments also introduce large but largely undocumented variations 
in task administration Evaluators may be able to obtain some of the needed 
information about this by means of interviews and questionnaires, but it may also be 

^In addition, teachers wilt often use different variants of common problems. For example, 
mathematics teachers in the 1993 scoring pointed out variants of a single problem used by 
different teachers that differed markedly in difficulty. 
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necessary to introduce planned variations into the operation of the assessment program 
to evaluate the quality of scores adequately? 

Measuring student performance. It is perhaf>s ironic that one of the most difficult 
problems in evaluating programs of this sort, which are themselves designed to measure 
student performance, is obtaining adequate measures of stiident performance. 

One reason that direct measures of student learning are needed is to gauge 
positive effects on learning. The new programs are typ^^ally intended at least in part to 
measure things than are not well tapped by extant assessments. Thus, even in 
jurisdictions that, unlike Vermont, have orbing assessments that will continue after the 
inception of the new program, there is a real possibility that trends on the old meas^ire 
will fail to register positive effects of the new program. (Moreover, scores on the old 
tests sometimes are not trustworthy as measures of trends on the skills they are 
supposed to assess. If the new programs lessen inappropriate teaching to the test on the 
old assessments, some decline in scores on the old test may be nothing more than the 
elimination of bias.) 

Additional direct measures of student performance are also critically important 
for validatioa They are needed to test the generalizability of performance even when 
the new programs are in their infancy, and they will become only more important over 
time as the possibility of inappropriate teaching to the test (and inappropriate 
administration) raises the specter of ir\flated scores on the new assessments. 

Unforttinately, there are many obstacles to obtaining sufficient direct measures of 
student performance. Here again, the pace of reform makes it difficult to obtain baseline 
measures. The costs of developing and admiiusteriitg the measures will also be an 
obstacle, as wiU access to schools (which are often understandably reluctant to allocate 
yet more time to testing). In addition, in Vermont, one of the most serious hindrances to 
independent measurement of student performance has been the insxifficient delineation 
of the domains that the new assessments are supposed to measure. As performance 
assessments in subjects other than writing become more contmoa we expect this 
problem to arise in many other programs as well. 

Hn Vermont, the open-ended questions in the uniform assessment should ideally provide 
some supplementary information. We have also proposed introducing standardized tasks (albeit 
with less than fully standardized administration) into the operation of the portfolio system itself. 



CONCLUSIONS 

The experience of the Vemont portfolio program to date suggests the need for 
nu}derate expectations, patience, and ongoing evaluation, not only in Vermont, but in 
other performance assessment programs as well. As Richard Mills and Ross Brewer 
acknowledged at the outset, the Vermont program (and, we add, many other 
performance assessment programs) will require a long period of development (Mills, 
and Brewer, 1988). Perhaps even more important, the Vermont experience illustrates the 
tensions among the goals of this and similar programs and the need to make difficult 
trade-offs in mediatii\g among them. Only time and careful scrutiny will show how 
fully the goals of the VermorU program-and of similar reform programs centered on 
perfomnance assessment-can be met, as well as what steps will need to be taken to meet 
them. 
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APPENDIX A. READER BIASES 



In both fourth and eighth grades, each writing portfolio was first rated by the 
student's own teacher; a sample was then re-scored by an independent reader. To 
explore any biases introduced by using the student's classroom teacher to rate his or her 
portfolio, we compared the mean score given by the classroom teacher to those given by 
the independent reader for the sample of portfolios scored twice. This comparison was 
made for both components of the portfolio (the best piece and the "rest") on each of the 
five scoring dimensions. 

The differences in means were generally very small, in most cases too small to be 
of any practical importance. For example, we foimd that in grade 4, the mean score 
assigned to pest pieces by students' own teachers on the "Purpose" dimension was 2.98; 
the corresponding mean for irKlependent readers was 2.93. With one exception, all of 
the other differences were less than O.L The means for the remaining components are 
given in Table A.l for grade 4 and Table A.2 for grade 8. 

With the exception of a single comparison (Voice/Tone for fourth-grade best 
pieces), none of the differences were statistically significant (usii^g a critical level of .05 
and not adjusting for multiple comparisons). T-statistics for all differences are shown in 
Tables A,l and A.2. The estimated mean scores and standard deviations were found 
using a jackknife technique (Cochran, 1977). jackknife estimates were used because 
students were sampled from schools. This clustering among students within schools 
may create a correlation among student scores from the same school. (That is, students 
from within a school may be more similar than randomly selected students.) The 
jackknife estimates make the appropriate adjmtment to standard errors to account for 
this clijstering. 

Also, readers could mark a portfolio or a piece of the portfolio as "non-scorable/' 
creating missing data for that case. If one reader identified a portfolio or piece as non- 
scorable but the other reader did not, the available score was used in estimatir^ the 
mean. Thus, a given portfolio could enter into one mean (the classroom teacher or 
independent reader) but not the other. Our results were not sensitive to this decision. 
Additional estimates were calculated by eliminating from both means all pieces 
identified as non-scorable by either reader, and these were the same as the means we 
used to two decimal places. 
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Table A.l 

Mean Seoraa for the Two Ratinae^ Grade 4 



"Ludcnc's Independent 

Teacher Reader 

Std std Mean 



Piece Dimension Mean Error Mean Error Pitt T* 



Best 


Purpose 


2 


.98 


0 


025 


2 


93 


0. 


026 


0 . 


OS 


1 . 


58 


Best 


Organization 


2 


.90 


0 


026 


2 


88 


0, 


027 


0, 


01 


0 . 


47 


Best 


Details 


2 


.81 


0 


028 


2 


76 


0. 


025 


0. 


05 


1 . 


4 7 


Best 


Voice/Tone 


2 


77 


0 


033 


2 


67 


0 . 


026 


0. 


10 




73 


Best 


Us age /Grammar 


2 


.84 


0 


027 


2 


87 


0 . 


02b 


-0 


.03 


-1 


.15 




/Mechanics 


























Rest 


F\jrpose 


2 


.85 


u 


026 


2 


79 


0 . 


029 


0. 


06 


1 . 


78 


Rest 


Organization 


2 


.74 


0 


027 


2 


.71 


0, 


030 


0 . 


04 


1 


02 


Rest 


Details 


2 


.59 


0 


.029 


2 


.57 


0 . 


031 


0 . 


02 


0 . 


57 


Rest 


Voice/Tone 


2 


.60 


0 


032 


2 


.54 


0 . 


028 


0 . 


07 


1, 


79 


Rest 


Usage /Grammar 


2 


.71 


0 


028 


2 


.70 


0 . 


028 


0 . 


01 


0 . 


48 




/Mechanics 



























NOTE: •T give the ratio of the difference in means to estimated the 
standard error of the difference. This statir;tic is approximately 
normal and can be compared to a standard Nornuil Table. 



Tabla h.2 

Maan Scoraa for the Two Ratinga^ Orada 8 



Student's Independent 
Teacher Reader 



Std Std Mean 

t'iece Dimension Mean Error Mean Error Diff T* 



Best 


Purpose 


3 


13 


U 


038 


3 


08 


0 


040 


0 . 


05 


1 . 


IB 


Best 


Organization 


3 


04 


0 


03 8 


2 


98 


0 


037 


0 . 


06 


1. 


56 


Rest 


Detai Is 




95 


0 


.043 


2 


95 


0 


042 


0 . 


00 


0 . 


04 


Best 


Voice/Tone 


3 


00 


0 


045 


2 


96 


0 


042 


0 . 


03 


0 . 


85 


Best 


Usage /Grammar 


2 


B3 


0 


039 


2 


.82 


0 


039 


0 . 


Ul 


0 . 


2V 




/Mechanics 


























Rest 


Purpose 


2 


92 


0 


.03 7 




.93 


0 


035 


-0 


.01 


-0 


.27 


HcsL 


Organ izat ion 


2 


S2 


0 


04 3 


2 


.80 


0 


04 0 


0 . 


02 


0 . 


48 


Rest 


Detailn 


2 


.70 


{) 


.039 


2 


.73 


0 


037 


-0 


. 03 


-{) 


.73 


Rent 


Voice/Tone 


2 


.79 


I) 


.04 3 


2 


.74 


0 


.04 3 


0 . 


0 4 


1 . 


04 


Rest 


Usage /Grammar 


7. 


.63 


0 


.036 


2 


.65 


') 


.036 


-{) 


. {)? 


-0 


.57 




/Mechanics 



























NOTK: ♦T give the ratio of the difference in means t^ estimated the 
:aandard error of the difference. This statistic is appj ox imately 
normal and can be compared to a standard Normal Table. 
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APPENDIX B 

Generalizability analysis investigates the dependability of person*s score on a 
single measure of ability. A dependable measure must generalize from a person's 
observed score to the average score the person would receive on all acceptable tests 
(Shave Ison and Webb, 1991). A given measure, e*g» a student's score from a single 
reader on single part of a portfolio, represer^ts performance on single realization of many 
alterable conditions. For example, the student could have produced a different part for 
the portfolio or a different reader could have scored the portfolio. Few people are 
interested in a stu^.ent's performance in a specific instance; rather, they are interested in 
the extent to which the student possesses the attribute that instance of measurement 
purports to assess. Thus, the meaningful score is not the student's score from a given 
reader on a given parts but the average over all scores the student would receive from 
all similar readers and similar parts* The score for given specific part and reader must 
generalize to this average over all parts and readers. These factors that one plans to 
generalize over are called facets (Shavelson and Webb, 1991). 

In the Vermont writing portfolios, each score depends on three factors, the 
student and the two generalizable facets, reader and part. Thus the score can be 
modeled as 

yijk = m 



+ 


mi - 


m 




(person effect) 


+ 


mj - 


m 




(reader effect) 


+ 


mk - 


m 




(part effect) 




mi j 


- mi 


- mj + m 


(person by reader effect) 


+ 


iHik 


- mi 


- mk + m 


(person by part effect) 


+ 


mjk 


- mj 


- mk + m 


(reader by part effect) 


+ 


Vijk 


- mi 


j - mik - mjk 






+ mi 


+ mj 


+ mk ~ m 


(residual effect) 



where i = 1 to the nximber of students {1741 for grade 4 and 655 for grade 8), j = 1 to the 
number of readers and k = 1 (best piece) or 2 (rest). Only the subset of student portfolios 
whose readers could be identified were included in this analysis. The procedure used 
for scoring student portfolios did not produce a fully crossed data set. Each portfolio 
was scored by only two readers, not by all readers (400 in grade 4 and 162 in grade 8). 
This creates an unbalanced data set. 
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In this study parts are "fixed." That is, we are interested in generalizing to any 
portfolio constructed with a best piece arvi the rtrai. Because part is fixed, no variance 
component will be estimated for it. The effects of students specific selection of pieces to 
be included as best or rest contributes to the student-by-part interaction. That is, the 
student-by-part interaction measures variability among the part scores from a single 
student. This variability may be the result of the student simply performing better on 
one of the parts or from the student's specific selection of pieces selection for both parts. 
These two sources of variability cannot be disentangled in our analyses. 

Our analysis only measures the extent of student item selection to the extent that 
the full range of possible pieces is represented in the items chosen to be in the portfolio. 
However, this is probably not a representative sample. Students were instructed to 
select their best work to be included in the portfolio. Thus, if Josh writes stories poorly, 
he may not have selected any stories for his portfolio. His portfolio scores would 
therefore not reflect the variability that would have been found if his portfolio had 
incltided stories. On the other hand, if Josh is weak in writing stories he would most 
likely never include a story in any portfolio he might construct. Therefore, the 
variability found in these portfolio scores may be representative of the variability that 
one would expect to find in scores from student-selected portfolio pieces. 

Genera lizability theory estimates the generalizability of a score by using the 
observed scores to estimate the variability of each effect or facet. For example, for the 
writing portfolios, we estimated the variability among different readers and the various 
interaction effects. We also estimated the variability among students. Student 
variability is considered the true measure of the variation in student ability. This 
variance is an estimate of the variability that exists in students' "universe " scores-the 
mean score a student would receive over all possible two-part self-selected portfolios 
scored using the current scoring procedure by all possible similar readers. The 
interpretation of the universe score is not clarified in generalizability analyses; it must be 
inferred from other sources such as validity studies. 

For each effect given in the above model, excluding the fixed part effect, we 
estimated the variability of that effect in the population of all such effects. These 
estimates indicate the amount of the variance in all possible portfolio part scores by all 
similar readers that is attributable to each effect. All effects other than the student effect 
are noise. That is, these effects are the results of the specific portfolio and reader and are 
not associated with the student's universe score. 
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Because of the unbalanced nature of the sample of scores, the traditional 
ANOVA-based estimates of components of variance were not available. Furthermore, 
the large sample size made it infeasible to use other ANOVA based methods to estimate 
the variance components. The component estimates were found using the MI VQUE 
estimation procedure (Hartley, Rao and LaMotte, 1978). This method produces 
unbiased, (locally) minimum variance estimates of the variance components. 

The estimates of the variances for the effects given in our model for student 
scores are in Table B.l for grade 4 arui B.2 for grade 8, The estimates were found 
separately for each dimer\sion. 

The generalizability of the portfolio score for a single dimer\sion is measured 
using the generalizability coefficient (Shavelson and Webb, 1991). The generalizability 
coefficient is approximately equal to the expected value (average) of the square of the 
correlation between the observed scores and the student's universe scores (Shavelson 
and Webb, 1991). It is also approximately equal to the correlation between observed 
scores on two analogous portfolios. For example, the generalizability coefficients given 
in Table B.l and B.2 are estimates of the correlation between two portfolios selected by 
the same student, each scored by a single reader who reads all pieces and assigns one 
score to the best piece and one score to the rest. This differs from the correlation 
coefficients given in Chapter 3 because this coefficient measures the degree of agreement 
between similar portfolios and readers simultaneously, rather than measuring 
agreement between readers on the same portfolio. 

The advantage of this generalizability study is that because we have estimated 
the components of variance, we can estimate the expected square of the correlation 
between universe scores and observed scores for various portfolio plans and scoring 
schemes. For example, we can estimate this correlation for a portfolio with three fixed 
parts, scored by two readers, where the score is the average of these two scores. The 
score from a portfolio is assumed to be the total (or average score) over all parts and 
readers for each dimension. 

These generalizability coefficients are meaningful only if we assume that the 
scoring procedure remair\s constant. That is, each reader scores all pieces together and 
then assigns part scores; the ir^dependent readers do r^t each score the parts separately 
or independently. We must assume the same scoring procedure because changit^ the 
scoring procedure might change the variability among scores. Also, this generalizability 
coefficient is only meaningful if we are considering similar parts constructed of similarly 
selected pieces. 
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OrAda 4 writing Variuca Coaponant Batiaataa 



Usage/ 
Grammar/ 

Purpose Organization Details Voice/Tone Mechanics 



Source 


Est. 


% 


Est . 


% 


Est. 


% 




% 


Est . 


% 


Student 


0.147 


27.5 


0.163 


28.7 


0.152 


26.5 


0.158 


23.8 


0.195 


32.7 


Student by 






















Parts 


0.032 


6.0 


0.040 


7.0 


0.042 


7.4 


0.042 


6.3 


0.056 


9.4 


Reader 


0.069 


12.9 


0.065 


11.5 


0.077 


13.5 


0.111 


16.7 


0.052 


8.8 


Reader by 






















Student 


0.007 


1.3 


0.006 


1.1 


0.009 


1.7 


0.007 


1.1 


0.006 


1.1 


Reader by 






















Parts 


0.085 


16.0 


0.090 


15.9 


0.110 


19.1 


0.137 


20.6 


0.115 


19.2 


Residual 


0.193 


36.2 


0.204 


35.9 


0.183 


31.9 


0.209 


31.4 


0.172 


28.8 


Gen. Coef. 


0. 


43 


0. 


43 


0. 


41 


0 , 


37 


0. 


46 



Tabla B.2 

Orada 8 Hriting Varianca Cooiponant Eatiaataa 



Usage/ 
Grammar/ 

Purpose Organization Details Voice/Tone Mechanics 



Source Est. % Est. % Est. % Est. % Est. % 



Student 


0 


.166 


31 


.1 


0. 


195 


33.1 


0. 


214 


35 


.4 


0. 


218 


33 


.3 


0 


281 


44 


.4 


Student by 








































Farts 


0 


.022 


4. 


2 


0. 


043 


7.2 


0. 


043 


7. 


2 


0. 


047 


7 . 


2 


0 


041 


6. 


4 


Reader 


0 


.047 


8. 


9 


0. 


047 


7.9 


0. 


046 


7. 


6 


0. 


066 


10 


.1 


0 


039 


6. 


1 


Reader by 








































Student 


0 


.008 


1. 


4 


0. 


008 


1 .4 


0. 


004 


0. 


7 


0. 


003 


0. 


4 


0 


006 


1 . 


0 


Reader by 








































Parts 


0 


. 107 


20 


.0 


0. 


122 


20.6 


0. 


103 


17 


.1 


0 . 


099 


15 


.1 


0 


104 


16 


.4 


Residual 


0 


.184 


34 


.4 


0. 


176 


29.7 


0. 


194 


32 


. 1 


0. 


221 


33 


.9 


0 


163 


25 


.7 


Gen. Coef. 




0. 


44 






0. 


46 




0. 


49 






0. 


48 






0. 


58 





- 107 - 



Under these assumptions, we can estimate the generalizability coefficient for any 
combination of parts and readers. The generalizability coefficients are calculated using 
the following formula 




where a]i = s, sr, sp and srp.e, denote the estimated variance components for 
students, student by reader, student by part and residual effects respectively, and nr and 
np denote the number readers scoring each portfolio and the number of parts, 
respectively. 

Typically the adjusted student-by-part effect would be included in the numerator 
as well as the denominator for fixed parts. However, although parts are fixed (best or 
rest), the pieces the students include in each part would change if the student were to 
construct an alternative portfolio. Thus the student-by-part interaction varies with 
portfolio and scoring, and this term belongs only in the denominator. For grade 4, 
Figures B.l to B,5 show the effects on the coefficient from varying the number of parts 
and the number of readers. Figures B.6 to B.IO are the analogous plots for grade 8. 
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APPENDIX C 



The generalizability study conducted with the math portfolio data is analogous 
to the study conducted on the writing portfolios. The facets of this study are piece and 
reader. In other words, the score a student receives on a given portfolio scored by a 
given reader must ger^ralize to the student's universe score from all portfolios 
constructed of similar pieces and scored by all similar readers. 

In the sample of piece scores each score depends on three factors, the student and 
the two generalizable facets^ reader and piece. Thus the score can be modeled as 



Yijk = \^ 

+ yijk - ^^ij - V^ik 



(person effect) 

(reader effect) 

(person by reader effect) 

(piece within person effect) 



(residual effect) 



where i ^ 1 to the number of students (803 for grade 4 and 355 for grade 8), j = 1 to the 
number of readers and k = 1 (best piece) or 2 (rest). Only the subset of student portfolios 
the readers of which could be identified were included in this analysis. The procedure 
used for scoring student portfolios did not produce a fully crossed data set. Each 
portfolio was scored by only two readers, not by all readers. This creates an unbalanced 
data set. 

In this study pieces are nested within the student. That is, there is no meaningful 
interpretation of pieces outside the students portfolio. In addition, piece number has no 
meaning. It is an arbitrary naming convention reflecting the order in which students 
entered pieces in their portfolios. Pieces could be randomly re-assigned different labels 
without changing the portfolio. This differs from writing, where pieces (or parts) were 
identified as either a best piece or the "rest" (the remaining pieces, which received a 
single score). Each piece in the math portfolio received a separate score on each of the 
seven scoring dimensions. These five to seven scores per dimension constitute a random 
sample of scores for each student. 

However, this is a random sample of scores for pieces the student might select 
and is probably not a representative sample of scores from all tasks the student might be 
asked to perform. Students were Instructed to select their best work to be included in 
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the portfolio. Thus, if Tiffany is weak at solving probability problems, she may not have 
included any such problems in her portfolio, and her portfolio scores do not indicate the 
variability that would result if she had. On the other hand, if Tiffany is weak in solving 
probability problems, she might never include such a problem in any portfolio she 
might construct. To the extent that is tru, the variability found in these portfolio scores 
would be representative of the variability that one would expect to find in scores from 
student-selected portfolios. 

Generalizability theory estimates the generalizability of a result by using the 
observed scores to estimate the variability of each effect or facet. For for the math 
portfolios, w^ estimated the variability among differer\t readers and the various 
interaction effects. We also estimated the variability among students. This is considered 
the true measure of the variation in student abilily. This variaiwe is an estimate of the 
variability that exists in students' "universe" scores— the mean score a student would 
receive over all possible five- to seven-piece self-selected portfolios scored using the 
current scoring procedure by all possible similar readers. The interpretation of the 
uruverse score, however, must be inferred from other sources siKh as validity studies. 

For each effect given in the above model we estimated the variability of that 
effect in the population of all such effects. These estimates indicate the amount of the 
variance in all possible portfolio piece scores by all similar readers that is attributable to 
each effect. All effects other than the student effect are noise. That is, these effects are 
the results of the specific portfolio and reader and are not associated with the student's 
universe score. 

Because of the unbalanced nature of the sample of scores, the traditional 
ANOVA-based estimates of components of varianca were not available. Furthermore, 
the large sample size made it infeasible to use other ANOVA-based methods to estimate 
the variance components. The compor^nt estimates were found using the MIVQUE 
estimation procedure (Hartley, Rao and LaMotte, 1978). This method produces and 
unbiased, (locally) minimum variance estimates of the variance components. 

The estimates of the variances for the effects given in our model for student 
scores are in Table C.l for grade 4 and C.2 for grade 8. The estimates are found 
separately for each dimension. 
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T^hl^ C.l 

Orad« 4 Hath Variance Coaponant SatiAAtae 





CI 




C2 




C3 








Source 


Est , 


% 


Est. 


% 


Est . 


% 






Student 


0.C'.03 


8.25 


0,0467 


7.14 


0.1480 


20.8/ 






Reader 


0.0743 


15.21 


0.0498 


7.62 


0.0729 


10.28 






Reader by 


















Student 


0.0313 


6.40 


0.0400 


6.13 


0 .0253 


3 ,57 






Piece 


















within 


















Reader 


0 .13122 


27 .04 


0.2348 


35.92 


0.1331 


18.78 






Residual 


0.2106 


43 .09 


0.2823 


43 .19 


0.3297 


46.50 








PSl 




PS2 




PS3 




PS4 






Est . 


% 


Est . 


% 


Est . 


% 


Est. 


% 


Student 


0.0505 


13 .06 


0 .0860 


14.82 


0.1571 


20.24 


0 .0211 


10.82 


Reader 


O.0181 


4.69 


0.0228 


3.93 


0 .1235 


15.92 


0.0136 


6.97 


Reader by 


















Student 


0.0258 


6.67 


0:0281 


4.85 


0.0213 


2.74 


0 ,0182 


9,30 


Piece 


















within 


















Student 


0 .0747 


19 .32 


0.1152 


19.85 


0.1197 


15.43 


0,0489 


25.01 


Residual 


0.2175 


56 .25 


0 .3282 


56.55 


0 .3543 


45, 67 


0.0936 


47 .90 



The generalizability of the portfolio score for a single dimension is measured 
using the generalizability coefficient (Shavelson and Webb, 1991), The generalizability 
coefficient is approximately equal to the expected value (average) of the square of the 
correlation between the observed scores and the student's universe scores (Shavelson 
and Webb, 1991). It is also approximately equal to the correlation between observed 
scores on two arulogous portfolios. For example, the generalizability coefficients given 
in Table C.l and C,2 are estiiiiates of the correlation between two portfolios selected by 
the same student, each scored by a single reader who reads all five pieces and assigns a 
score to each pieoe. This differs from the correlation coefficients given in Section Y 
because this ooelfficient measures the degree of agreement between similar portfolios 
and readers simultaneously, rather than measuring agreement between readers on the 
same portfolio. 
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Grade B Math Variaaea Componant Eatimataa 





CI 




C2 




C3 






Souirce 


Est . 


% 




ft 


ILSU . 










11./ 


U . UJ33 


c no 


0 . 1849 


22.80 




Reader 


n n c 1 Q 
U - U5 1 y 


10.4 


0 .0523 


7 . 58 


0 . 0454 


5 .60 




















scuaen u 


u , udd? 


11.2 


0 • 0 JQl 


4.36 


0 . 0708 


8 . 73 




Piece 
















wi t Hin 
















Reader 


0 .1197 


24 .1 


0 .2827 


4 0 . 99 


n 1 ft 1 n 


5 0 lb 




Residual 


0.2115 


42.5 


0.2847 


41.28 


0.3289 


4 0.56 






PSl 




PS2 




PS3 




PS4 




Est . 


% 


Est. 


% 


Est. 


% 


Est. % 


Student 


0.0435 


11.59 


0 .0526 


9.70 


0.1117 


14.1 


0.0153 10.59 


Reader 


0.0103 


2 .75 


0 .0236 


4.35 


0.0912 


11.5 


0.0071 4.90 


Reader by 
















Student 


0.0363 


9 .69 


0 .0469 


8.64 


0.0829 


10.5 


0.0019 1.32 


Piece 
















within 
















Student 


0.0884 


23.59 


0 .1544 


28.48 


0.1858 


23.4 


0.0345 23.83 


Residual 


0.1965 


52 . 38 


0.2648 


48.83 


0.3213 


40.5 


0.0859 59.36 



The advantage of this generalizability analysis is that because we have estimated 
the components of variance, we can estimate the expected square of the correlation 
between universe scores and observed scores for various portfolio plans and scoring 
schemes. For example, we can estimate this correlation for a portfolio with five pieces, 
scored by two readers, where the score is the average of these ten scores. The score from 
a portfolio is assumed to be the total (or average score) overall parts and readers for each 
dimension. 

These generalizability coefficients are meaningful only if we assume that the 
scoring procedure remains constant. That is, each reader scores all pieces together; the 
independent readers do not each score the parts separately or independently. We must 
assume the same scoring procedure because changing the scoring procedure might 
change the variability among scores. Also, this generalizability coefficient is only 
meaningful if we are considering similar selected pieces. 

Under these assumptions, we can estimate the generalizability coefficient for any 
combination of parts and readers. The generalizability coefficients are calculated using 
the following formula 
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where dj i = s, er, p and srp.e, denote the estimated variance components for 
students, student by reader, piece within student and residual effects respectively, and 
nr and np denote the number readers scoring each portfolio and the number of pieces, 
respectively. 

For grade 4, Figures C.l to C.7 show the effects on the coefficient from varying 
the number of parts and the number of readers. Figures C.8 to C.14 are the analogous 
plots for grade 8. 
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